CARTAN-DECOMPOSITION SUBGROUPS OF S0(2,n) 
HEE OH AND DAVE WITTE 

Q> I Abstract. For G = SL(3,R) and G — SO(2,n), we give explicit, practical conditions that de- 

0^ ' termine whether or not a closed, connected subgroup H of G has the property that there exists a 

0^ ' compact subset C of G with CHC = G. To do this, we fix a Cartan decomposition G = KA^K 

of G, and then carry out an approximate calculation of (KHK) D A'^ for each closed, connected 

5h ■ subgroup H of G. 
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1. Introduction 



f-H I 1.1. Notation. Throughout this paper, G is a Zariski-connected, almost simple, linear, real Lie 

P^ ■ group. ("Almost simple" means that every proper normal subgroup of G either is finite or has finite 

(-H , index.) In almost all of the main results, G is assumed to be either SL(3, R) or S0(2, n) (with n > 3). 

There would be no essential loss of generality if one were to require G to be connected, instead of 

only Zariski connected (see 3.2l| ). However, S0(2,n) is not connected (it has two components) and 



the authors prefer to state results for S0(2,n), instead of for the identity component of S0(2,n). 

Fix an Iwasawa decomposition G = KAN and a corresponding Cartan decomposition G = 

KA^K, where A^ is the (closed) positive Weyl chamber of A in which the roots occurring in the 

Q"^ \ Lie algebra of N are positive. Thus, K \s a, maximal compact subgroup, A is the identity component 

■^ ' of a maximal split torus, and A^ is a maximal unipotent subgroup. 

O , 

CN \ The terminology introduced in the following definition is new, but the underlying concept is well 

^^ ■ known (see, for example, Proposition 1.4 and Theorem |1.6| below). 

5^ ! 1-2. Definition. Let H he a. closed subgroup of G. We say that H \a a, Cartan- decomposition 

subgroup of G if 

J^ ■ • if is connected, and 

C ■ • there is a compact subset G of G, such that GHC = G. 

(Note that G is only assumed to be a subset of G; it need not be a sMhgroup.) 

• 1-^ - 

>< 
^^ ■ 1.3. Example. The Cartan decomposition G = KAK shows that the maximal split torus ^ is a 

Cartan-decomposition subgroup of G. 

It is known that G = KNK |Ko£, Thm. 5.1], so the maximal unipotent subgroup N is also a 



Cartan-decomposition subgroup. 

If G is compact (that is, if M-rankG = 0), then all subgroups of G are Cartan-decomposition 
subgroups. On the other hand, if G is noncompact, then not all subgroups are Cartan-decomposition 
subgroups, because it is obvious that every Cartan-decomposition subgroup of G is noncompact. 
It is somewhat less obvious that if ii is a Cartan-decomposition subgroup of G, then dim/i > 



^rankG (see ^jj). 



Our interest in Cartan-decomposition subgroups is largely motivated by the following basic ob- 
servation that, to construct nicely behaved actions on homogeneous spaces, one must find subgroups 
that are not Cartan-decomposition subgroups. (See | Kb3| , §3] for some historical background on 
this result.) 
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1.4. Proposition (Calabi-Markus phenomenon, cf. |Ku], pf. of Thm. A. 1.2]). If H is a Cartan- 
decomposition subgroup of G, then no closed, noncompact subgroup of G acts properly on G/H. 

Our goal is to determine which closed, connected subgroups of G are Cartan-decomposition 
subgroups, and which are not. Our main tool is the Cartan projection. 

1.5. Definition (Cartan projection). For each element g of G, the Cartan decomposition G = 
KA^K implies that there is an element a of A~^ with g G KaK. In fact, the element a is unique, 
so there is a weU-defined function // : G ^ A'^ given hy g € K n(g) K. The function /i is continuous 
and proper (that is, the inverse image of any compact set is compact). Some properties of the 
Cartan projection are discussed in pen| ] and |KbS]. 



We have /u(ff) = A'^ if and only if KHK = G. This immediately implies that if fJ,{H) = A^, 
then H is a Cartan-decomposition subgroup. Y. Benoist and T. Kobayashi proved the deeper 
statement that, in the general case, H is a Cartan-decomposition subgroup if and only if ^{H) 
comes within a bounded distance of every point in A'^ . 



1.6. Theorem (Benoist [Ben, Prop. 5.1], Kobayashi jKb2 , Thm. 1.1]). A closed, connected sub- 
group H of G is a Cartan-decomposition subgroup if and only if there is a compact subset G of A, 
such that ^i{H)C D A+ . 

We noted above that every subgroup is a Cartan-decomposition subgroup if M-rankG = 0. 
Therefore, the characterization of Cartan-decomposition subgroups of G is trivial if M-rankG = 0. 
The following simple proposition shows that the characterization is again very easy if M-rankG = 1. 



1.7. Proposition (cf. [ Kbl , Lem. 3.2]). Assume that R-rankG = 1. A closed, connected sub- 
group H of G is a Cartan-decomposition subgroup if and only if H is noncompact. 

Proof. (<^) We have /x(e) = e, and, because /x is a proper map, we have /x(/i) ^^ oo as /i — s- oo in ff. 
Because M-rankG = 1, we know that A'^ is homeomorphic to the half-line [0, oo) (with the point e 
in A'^ corresponding to the endpoint of the half- line), so, by continuity, it must be the case that 
^{H) = A'^ . Therefore KHK = G, so H is a Cartan-decomposition subgroup. D 

It seems to be much more difficult to characterize the Cartan-decomposition subgroups when 
M-rankG = 2, so these are the first interesting cases. In this paper, we study two examples in detail. 
Namely, we describe all the Cartan-decomposition subgroups of SL(3,M) and of S0(2,n). We also 
explicitly describe the closed, connected subgroups that are not Cartan-decomposition subgroups, 
and approximately calculate the image of each of these subgroups under the Cartan projection. 

Obviously, any connected, closed subgroup that contains a Cartan-decomposition subgroup is 
itself a Cartan-decomposition subgroup. Therefore the minimal Cartan-decomposition subgroups 
are the most interesting ones. As a simple example of our results, we state the following theorem. 

1.8. Theorem. Assume that G = SL(3,M). Up to conjugation by automorphisms of G, the only 
minimal Cartan-decomposition subgroups of G are: 




A, 

and subgroups of the form 
(1.9) 



r, s G 



' pP f 





e^* 

e-(P+'?)* 
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where p and q are fixed real numbers with max{p, q} = 1 and min{p, q} > —1/2, or of the form 



(1.10) 



e cos pt e sin pt 
—e^ sin pt e^ cos pt 



-2t 



r,s,t £ 



e" 

where p is a fixed nonzero real number. 

Note that AN contains uncountably many nonconjugate minimal Cartan-decomposition sub- 



groups of G, because the minimum of the two parameters p and q in (1.9) can be varied contin- 



uously. However, up to conjugacy under Aut G, there is only one minimal Cartan-decomposition 
subgroup contained in A (namely, A itself), and only one contained in N. 

1.11. Corollary. Let H be a closed, connected subgroup of G = SL(3, M), and let Kjj be a maximal 
compact subgroup of H . If dim H — dim K h > 3, then H is a Cartan-decomposition subgroup of G. 

For an explicit description of the Cartan-decomposition subgroups of SL(3,M) (not just up to 
conjugacy), see Theorem ^. The subgroups of SL(3,M) that are not Cartan-decomposition sub- 



groups are described in Corollary 4.7, and their images under the Cartan projection are described 



in Proposition [4.I4 These results are stated only for subgroups of AN ., because the general case 
reduces to this (see Remark 2.10| ). 

Theorem 1.14 is a sample of our results on Cartan-decomposition subgroups of S0(2,n). Note 
that, for simplicity, we restrict here to subgroups of N . 



1.12. Notation. We realize S0(2, n) as isometries of the indefinite form {v \v) = viVn+2-\-V2Vn+i + 
X]r=3 ^i^ o^ M'^^^ (for V = (fi,f2, . . . ■,Vn+2) G M*^"*"^). The virtue of this particular realization is 
that we may choose A to consist of the diagonal matrices in S0(2,n) (with nonnegative entries) 
and A'^ to consist of the upper-triangular matrices in S0(2,n) with only I's on the diagonal. Thus, 
the Lie algebra of ^A'^ is 



(1.13) 



a-|- n 



r Ai 



v 



t2 



y 
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-t2 
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-hj 



ti,t2,(p,7j£U 

x,y eM'^-^ 



Note that the first two rows of any element of a + n are sufficient to determine the entire matrix. 

Whenever m < n, there is an obvious embedding of S0(2, m) in S0(2, n), induced by an inclusion 
]^m+2 ^^ i^"+2^ sQ^ abusing notation, we speak of S0(2,m) as a subgroup of S0(2,n). 

1.14. Theorem. Assume that G = S0(2,n). If n > 5, then there are exactly 6 non-conjugate 
minimal Cartan-decomposition subgroups of G contained in N . Each such subgroup H is conjugate 
to a subgroup o/SO(2,5) and, as a subalgebra 0/50(2, 5), the Lie algebra of H is conjugate to one 
of the following: 

^0 ?7 
1) < I ei(/) -r/ I (/>,?? G M V, w/iere ei G {0, 1} 



[ /O X 0^ 

2) < e20 

^0 a; esy 0^ 

3) ^ ( y 



\x eM. ) , where €2 G {0,1} 
X, y G M ^ , where €3 G {0,1}. 
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There are 5 non-conjugate minimal Cartan- decomposition subgroups o/SO(2,4) contained in N . 
The Lie algebra of any such subgroup H is conjugate either to one of the two subalgebras of type (||), 
to one of the two subalgebras of type (|^, or to the subalgebra of type @ with £3 = 0. {These are the 
five of the above-listed subalgebras that are contained mso(2,4), namely, the five whose 5th column 
is all O's.) 

There are 3 non-conjugate minimal Cartan- decomposition subgroups o/SO(2,3) contained in N. 
The Lie algebra of any such subgroup H is conjugate either to one of the two subalgebras of type (|^), 
or to the subalgebra of type (^ with ei = 0. [These are the three of the above-listed subalgebras that 
are contained in 5o(2,3), namely, the three whose 4th and 5th columns are all O's.) 

The detailed study of Cartan-decomposition subgroups of S0(2, n) is rather compUcated, so we 
break it up into three parts: subgroups of N (Theorem |5.3[), subgroups not in N that can be written 



as a semidirect product T t< U with T <Z A and U d N (Theorem xl), and subgroups that cannot 



be written as such a semidirect product (Theorem |6.3| ). We also describe the subgroups of S0(2, n) 



that are not Cartan-decomposition subgroups (see Theorem 5^ and Corollaries ^^ and |6.4D , and 
approximately calculate their Cartan projections (see Proposition 5^ and Corollaries 6^ and |6.4| ). 
If ff is a Cartan-decomposition subgroup of G, and G/H is not compact, then the Calabi- 
Markus phenomenon |1.4| implies that G/H does not have a compact Clifford-Klein form. (That 
is, there does not exist a discrete subgroup F of G that acts properly on G/H, such that the 
quotient space T\G/H is compact.) Thus, our work on Cartan-decomposition subgroups is a first 
step toward understanding which homogeneous spaces of G have a compact Clifford-Klein form. 
Building on this, a sequel [p"W | determines exactly which homogeneous spaces of S0(2,n) have a 



compact Clifford-Klein form in the case where n is even (and assuming that the isotropy group H is 
connected), but the results are not quite complete when n is odd. The work leads to new examples 
of compact Clifford-Klein forms of S0(2,n), when n is even. 

The paper is organized as follows. Section |2| collects some known results on Lie groups and Zariski 
closures. Section ^ presents some general results on Cartan-decomposition subgroups. Section ^ 
states and proves our results on Cartan-decomposition subgroups of SL(3,M). Section ^ contains 
our results on Cartan-decomposition subgroups of S0(2, n) that are contained in N, and Section |6| 
is devoted to the subgroups of S0(2, n) that are not contained in N. 

1.15. Acknowledgment. This research was partially supported by grants from the National Science 
Foundation (DMS-9623256 and DMS-9801136). Many of the results of this paper were finalized 
during a visit of the authors to the University of Bielefeld. We would like to thank the German- 
Israeli Foundation for Research and Development for financial support that made the visit possible, 
and the mathematics faculty of the University of Bielefeld for their hospitality that helped make 
the visit so productive. We are also grateful to Kari Vilonen for explaining that the theory of 



o-minimal structures would be useful to us (see 3.13 and 3.14 ). D. Witte would like to thank the 



Tata Institute for Fundamental Research for providing a congenial environment to carry out final 
revisions on the manuscript. 

2. Preliminaries on Lie groups and Zariski closures 

Most of the results in this section are well known, and none are new. The reader is encouraged 
to skip over this section, and refer back when necessary. 

We assume familiarity with the basic theory of Lie groups and Lie algebras (as in, for example, 
[ Hoc(1 ). At some points, we also assume some familiarity with the structure of algebraic groups 



over M, in the spirit of [Rag, §P.2, pp. 7-11]. 



2.1. Notation. We use German letters g, f), a, n, u, t to denote the Lie algebras of Lie groups G, 
H, A, N, U, T, etc. 
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For a linear functional to on a, we use u^^ to denote the corresponding weight space of the Lie 
algebra 3 (so u^^ = unless oj is either or a real root of G), and U^^ to denote expu^^. Note that 
Uu)U2uj is a subgroup of G (but U^) may not be a subgroup if both uj and 2uj are real roots of G). 

The idea of the following definition is to require that the choice of the Cartan subgroup A be 



compatible with a particular subgroup H . It is not a severe restriction on if, because Lemma ^ 
shows that it can always be satisfied by replacing H with a conjugate. Then, under the assumption 
that M-rankG = 2, Lemma |2.4] shows that H has a fairly simple description in terms of (1) an 
appropriate subgroup of A, (2) the intersection of H with A^, and, perhaps, (3) a homomorphism 
into a root subgroup of G. (If M-rankG > 2, then the description of a typical H would require 
several root subgroups.) 

2.2. Definition. Let us say that a subgroup H of AN is compatible with A \i H d TUG]\[{T), 
where T = AO (HN), U = H f] N, and Cn{T) denotes the centralizer of T in N . 

2.3. Lemma. If H is a closed, connected subgroup of AN, then H is conjugate, via an element 
of N, to a subgroup that is compatible with A. 

Proof. Let H be the identity component of the Zariski closure of H, and write H = T kU , where U 
is a subgroup of N and T is conjugate, via an element of A^, to a subgroup of A. Replacing if by a 
conjugate, we may assume that T d A. LetU = HCiN. Then [H,7i] C HnN (cf. f7s^, Cor. 3.8.4, 



p. 207] and | Wit , Lem 3.24]), so we have [T,u\ C u. Because T C A and u is (AdG(T))-invariant, 
the adjoint action of T on u is completely reducible, so this implies that there is a subspace c of u, 
such that [T, c] = and u + c = u. Therefore, U Cn(T) = U,soH = TUCn(T). _ 

Let vr: AN — > A be the projection with kernel iV, and let T = tt{H). Then T = Tr{H) C vr(ii) = 
T, so C]\f{T) D Cn{T). For any h £ H, there exist t gT, u € U and c G CNiT), such that h = tuc. 
Because uc G A^, we must have t = 7r(/i) G T and, because C]y(T) D Cn{T), we have c G Cn{T). 
Therefore, h G TUGn{T). We conclude that H C TUGn{T), so H is compatible with A. D 

2.4. Lemma. Assume that M-rankG = 2. Let H be a closed, connected subgroup of AN , and 
assume that H is compatible with A. Then either 

1) H = {HnA)tK (HnN); or 

2) there is a positive root uj, a nontrivial group homomorphism ip: keroj -^ U^U2u), o^nd a closed, 
connected subgroup U of N , such that 

(a) H = { ailj{a) \ a G ker w }U ; 

(b) U n ^(kerw) = e; and 

(c) U is normalized by both kerw and ijj{ke,xijj). 

Proof. Because H is compatible with A, we have H C TUG]y{T), where T = A (1 (HN) and 
U = H n N. We may assume that H / TU, for otherwise we have H = {H n A) t< {H DN). 
Therefore Gm{T) 7^ e. Because n is a sum of root spaces, this implies that there is a positive 
root io, such that T C kerw. Because R-rankG = 2, we must have T = kertij, for otherwise we 
would have T = e, so H = U = TU. Therefore, Cn{T) = UujU2lj. 

Because U C H C TUGn{T), we have H = U\H r\ (rCAr(r))], so there is a nontrivial one- 
parameter subgroup {x*} in Hr\(TGN{T)) that is not contained in U. Because T centralizes Gn{T), 
we may write x* = a*u* where {a*} is a one-parameter subgroup of T and {n*} is a one-parameter 
subgroup of Cn{T). Furthermore, this Jordan decomposition is unique, because T n Gn{T) = e. 
Replacing ii by a conjugate subgroup, we may assume that a* G A. Define ^■. keruj — > Ui^U2uj by 
^'(a*) = It* for all t G M. 

( pa] ) For all i G M, we have a^ip{a^) = a^u^ = x* G if, which establishes one inclusion of (|2a|). 
The other will follow if we show that dim if — dimf/ = 1, so suppose dim if — dimC/ > 2. Then 
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Lemma 2.8 implies that A C H, so it follows from Lemma 2.5 that H = AtK (HnN), contradicting 
our assumption that H ^ TU. 

( pb| ) Suppose U n tlj{keTuj) ^ e. Because the exponential map from n to A^ is bijective, we 
know that the intersection is connected. Because dim(keru;) = 1, this implies that ^(kerw) C U. 
Therefore a* = x^u"* G HU = H,soT C H. This contradicts the fact that H / TU. 

( p^ Because x* G H, we know that the Jordan components a* and it* of x* belong to the Zariski 
closure of H |Hm2, Thm. 15.3, p. 99]. Therefore, both of a* and u* normalize H (see 2.6). Being 



in AN, they also normalize N. Therefore, they normalize H n N = U. D 



2.5. Lemma (cf. |Hml , pf. of Thm. 20.2(d), pp. 108-109]). If H is a dosed connected subgroup 



of AN that is normalized by A, and u> is a weight of the adjoint representation of A on a + n, then 
TTuji^i)) C f), where vr^ : a + n — > u^; is the A-equivariant projection. 
In particular, letting uj = 0, we see that H = (H f] A) t< (H n N). 

2.6. Lemma (cf. |Zim| , pf. of Thm. 3.2.5, p. 42]). If H is a closed, connected subgroup ofG, then 
the Zariski closure of H normalizes H. 



2.7. Lemma (cf. |Bor, Thm. 10.6, pp. 137-138]). If H is a closed connected subgroup of AN, such 
that H has finite index in its Zariski closure, then H can be written as a semidirect product H = 
T \kU , where U is a subgroup of N and T is conjugate, via an element of N , to a subgroup of A. 

2.8. Lemma. Let H be a closed, connected subgroup of AN . If dim. H — dini{H f] N) > M-rankG, 
then H contains a conjugate of A, so H is a Cartan- decomposition subgroup. 

Proof. Let vr: AN — > A be the projection with kernel A^. Since 

M-rankG < (dimH) - dim{H n N) = dim(7r(i7)) < dim^ = M-rankG, 

we must have it{H) = A, so HN = AN. Therefore, letting H be the Zariski closure of H, we may 



assume that H contains A (see 2.7), by replacing H with a conjugate subgroup. So H is normalized 



by A (see p.Gp. Therefore, since vr(i?) = A, we conclude that A C H (see 2.5). D 



All maximal compact subgroups of any connected Lie group are conjugate [ Hoc , Thm. XV. 3.1, 
p. 180-181], so the quantity dim i? — dim X/f in the statement of the following lemma is independent 
of the choice of Kh • 

2.9. Lemma. Let H be a closed, connected subgroup of G. Then there is a closed, connected 
subgroup H' of G and a compact subgroup C of G, such that CH = CH' , and H' is conjugate to 
a subgroup of AN . Furthermore, there is a continuous function f : H ^ H' with f{h) G Ch for 
every h a H , and we have diuiH' = diuiH — dim Kh, where Kh is a maximal compact subgroup 
ofH. 

Sketch of proof. Let L be a maximal connected semisimple subgroup of H, and let T be a maximal 
compact torus of the Zariski closure of Rad H, the solvable radical of H. Replacing T by a conjugate 
torus, we may assume that L centralizes T. Let L = K^AlNl be the Iwasawa decomposition 
of L. From L. Auslander's nilshadow construction (cf. |Wit| , §4]), we know that there is a unique 
connected, closed subgroup R of G, such that R is conjugate to a subgroup of AN, and RT = 
(Rad H)T . Let H' = A^N^R, and note that the uniqueness of R implies that H normalizes R, so 
H' is a subgroup of G. Then KlTH' = TH = KlTH. 

Define f : H ^> H' hy specifying that h G KlT ■ f{h). Because H' is conjugate to a subgroup 
of AN, which has no nontrivial compact subgroups, we have {K^T) D H' = e, so f{h) is well 
defined. D 
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2.10. Remark. The proof of Lemma |2.9| is constructive. Furthermore, given a subgroup H and the 
corresponding subgroup H' , it clear that H is a Cartan-decomposition subgroup if and only if H' is 
a Cartan-decomposition subgroup. Thus, to characterize all the Cartan-decomposition subgroups 
of G, it suffices to find all the Cartan-decomposition subgroups that are contained in AN. 

2.11. Remark. Our restriction to closed subgroups in the definition of Cartan-decomposition sub- 
groups is not very important. Namely, if one were to allow non-closed subgroups, then one would 
prove that a subgroup is a Cartan-decomposition subgroup if and only if its closure is a Cartan- 
decomposition subgroup. This follows from the theorem of M. Goto and A. Malcev (independently) 
that if i? is a connected Lie subgroup of G, then there is a compact subgroup G of G, such that CH 



is the closure of H [Pog, Thm. 1.3]. (This theorem can be derived from the proof of Lemma 



because every connected subgroup of AN is closed.) 

3. General results on Cartan-decomposition subgroups 

3.1. Notation. We employ the usual Big Oh and little oh notation: for functions /i,/2 on H, 
and a subset Z of H, we say /i = 0{f2) for z & Z if there is a constant C, such that, for all large 
z £ Z, we have ||/i(2;)|| < (7||/2(z)||. (The values of each /j are assumed to belong to some finite- 
dimensional normed vector space, typically either M or a space of real matrices. Which particular 
norm is used does not matter, because all norms are equivalent up to a bounded factor.) We say 
/i = o(/2) forz£Z if 11/1(2)11/11/2(2)11 ^ as z ^ 00. Also, we write /i x /a if /i = ©(/a) and 
/2 = 0(/i). 



3.2. Notation. It is known (cf. [Hen, Lem. 2.3]) that there exist irreducible real finite-dimensional 
representations pi, i = 1, . . . , fc of (a finite cover of) G, such that the highest weight space of each pi 
is one-dimensional and if Xi is the highest weight of pi , then {xi \ i = ^,' ■ ■ ,k} is a basis of the 
vector space A* of all continuous group homomorphisms from A to M"*" (where the vector-space 
structure on A* is defined by {sa + t(5){a) = a(a)^/3(a)* for s,t G R, a, /? G A*, and a G A). In 
particular, we have k = M-rankG. 

When we have fixed a particular choice of /Jj, i = 1, . . . , A;, we may refer to pi, . . . , p^ as the 
fundamental representations of G. 



3.3. Proposition (Benoist |Ben , Lem. 2.4]). For each i = 1, . . . ,M-rankG, we have Xi(/w(5)) ^ 
Piig), for g G G. 

Because p{g) is determined by the values xi (/"(<?)))■■ • ■,Xk{l^{9)) (and Xi[o) = IXi(o^)l for each 
a G ^), it follows from the preceding proposition that the Cartan projection p[g) can be calculated 
with bounded error by finding the norms of pi{g),... ,pk{g). (The error bound depends only 



on G; it is independent of g). This is theoretically useful (see, for example. Corollaries 3.4, 



and 3.7, and note that Theorem |1.6| is a special case of Corollary pA) and is also the method we 
use in practice in Sections §, ^ and|6| to calculate the image under // of subgroups of SL(3, M) and 
S0(2,n). 

3.4. Corollary (Benoist ]Ben] , Prop. 5.1]). For any compact set Gi in G, there is a compact set G2 
in A, such that p{GigGi) C p{g)G2, for all g £ G. 



3.5. Corollary (Benoist jBen] , Prop. 1.5], Kobayashi ]Kb2, Cor. 3.5]). Let Hi and H2 be closed 



subgroups of G. The subgroup Hi acts properly on H2 if and only if, for every compact subset G 
of A, the intersection (^p{Hi)G) Ci p{H2) is compact. 

3.6. Notation. For subsets X,Y C A~^ , we write X ~ y if there is a compact subset C of ^ with 
X C YG and Y C XG. 
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It is obvious from the definition that if i:f is a Cartan-decomposition subgroup, then every 
conjugate of H is also a Cartan-decomposition subgroup. In other words, if ^{H) ~ A"*", then 
fi{g~^Hg) ~ A~^. The following corollary is a generalization of this observation. 

3.7. Corollary. If H is a subgroup of G, then fi{g~^Hg) f« fJ-{H), for every g £ G. 

In particular, every conjugate of a Cartan-decomposition subgroup is a Cartan-decomposition 
subgroup. 

3.8. Notation. Suppose that M-rankG = 2 and let {ai, 02} be the set of simple roots with respect 
to A'^ in G. We denote by L, the wall of A~^ defined by Oj = 1 for each i = 1,2. Since, by definition, 
{xi)X2} is a basis of A*, there exists some real number ki such that 

Li = {aeA+ \xiia)''' =X2{a)}. 

(Note that Oi cannot be a scalar multiple of xi? because xi) being a highest weight, is in A'^, but 
a, being a simple root, is not.) Although we do not need this general fact, we mention that ki is 
always a rational number. 

We now introduce convenient notation for describing the image of a subgroup under the Cartan 
projection fi. 

3.9. Notation. Suppose that p is a representation of G and that G is a matrix group (that is, 
G C GL(i, M), for some i), and assume that the two fundamental representations of G are pi{g) = g 



and P2{g) = p{g) (cf. Notation 3^). For functions fi,f2' 1^^ -^ 1^"'', and a subgroup H of G, we 



write p{II) ^ [fi{\\h\\), /2(||^||)] if> for every sufficiently large C > 1, we have 
^,{H) ^{aeA+\ C-Vi(l|a||) < Ma)\\ < Gf2{\\a\\) } . 
(If /i and /2 are monomials, or other very tame functions, then it does not matter which particular 



norm is used.) In particular, from Proposition 3.3, we see that H is a. Cartan-decomposition 



subgroup of G if and only if p{H) ~ [||/i||^, ||/i||^] , where ki and k2 are as described in Notation 



If M-rankG = 2, Proposition ^.14 provides a simple way to determine whether or not a sub- 



group H of ^A^ is a Cartan-decomposition subgroup. One simply needs to determine whether 
or not there are arbitrarily large elements hi and /12 of H, such that ||p2(^i)|| is approximately 

\\Piihi)\\'^. 

The proof of Proposition |3.14|uses some basic properties of polynomials of exponentials. The class 
of exp-definable functions ( see |3.1C| ) is much more general, and the theory of the real exponential 
function is o- minimal [|W1| , [W2| ], so general properties of o- minimal structures |PS, KPS| , vdD| tell 



us that every exp-definable function behaves very much like an ordinary polynomial. 
The following definition assumes some familiarity with first-order logic. 

3.10. Definition (cf. [|vdD| , (5.3)]). A subset X of M" is ex.p- definable if there are real numbers 
oi, . . . , Om, and a first-order sentence 4>{xi, . . . , a;„, yi, . . . , ym) in the language of ordered fields 
augmented with the exponential function exp, such that 

X = { (xi,... ,Xn) G K" I 0(xi,... ,Xn,ai,... ,am)}- 

A map f : X ^ Y is exp-definable if its graph is an exp-definable subset of X x y. 



We remark that every exp-definable set X has a well-defined dimension vdD , p. 5 and p. 63]. 
Namely, X can be written as the union of finitely many cells, and the dimension of X is the 
maximum of the dimensions of these cells. 

3.11. Lemma. If H is a closed, connected subgroup of AN , then p{II) is an exp-definable subset 
of A, and dim.p{H) < dimH. 
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Proof. Because H = exp f), it is easy to see that H is an exp-definable subset of G. Furthermore, the 
map fj, is obviously exp-definable. Therefore, fJ-{H) is an exp-definable subset of A, and diinfi(H) < 
dimH ^dg. Cor. 4.1.6(ii), p. 66]. D 



3.12. Lemma ([vdD, Props. 3.2.8 and 6.3.2, pp. 57, 100]). Any exp-definable set has only finitely 
many path- connected components. 



3.13. Proposition. Let H be a closed, connected subgroup ofG, and letKfj be a maximal compact 
subgroup of H . If H is a Cartan- decomposition subgroup of G, or, more generally, if no closed, 
noncompact subgroup of G acts properly on G/H, then dim/7 — dimi^/^ > M-rankG. 

Proof. We may assume that H C AN 
contrapositive: suppose that dimff < 



(see p^ ), so divnH — (MuiKh = divuH. We prove the 
R-r ank G . Let s = {a G o+ | exp(a) e /u(-ff) }• Then 
s is exp-definable and dims < dimff (see 3.11 ). Let ttoo be the sphere at infinity, and let Sqo 
be the set of accumulation points of s in aoo. For any nonempty exp-definable set S, we have 



divudS < dimS", where dS is the complement of S in its closure [ ydP , Thm. 4.1.8, p. 67], so we 
know that dimSoo < dims. Therefore, dimSoo < dimo — 1 = dim aoo, so Sqo does not contain any 
nonempty open subset of Ooo. On the other hand, it is obvious that a+ does contain an open subset 
of Ooo, so we conclude that some point r G Oqo does not belong to Sqo- Let r+ be the ray in a"*" from 
in the direction r. Because r ^ Sqo, we know that, for every compact subset c of a, the intersection 
r"*" n (s + c) is bounded. Thus, Corollary 3^ implies that, for every compact subset G of G, the 
intersection exp(r+) n GHG is bounded. Therefore, the one-parameter subgroup exp(r^ U — r^) 
acts properly on G/H, so H is not a Cartan-decomposition subgroup (see |1.4|). D 



The converse of Proposition 1.4 is not known to hold in general. However, the following propo- 
sition (combined with Proposition |1.7| ) shows that the converse does hold if M-rankG < 2. Condi- 
tion |3.14| (p|) is used throughout Sections ^ |5|, and ^ to determine whether or not a subgroup of G 
is a Cartan-decomposition subgroup. 

3.14. Proposition. Assume that M-rankG = 2. If H is a connected subgroup of AN , then the 
following are equivalent: 

1) H is a Cartan-decomposition subgroup of G. 

2) No closed, noncompact subgroup of G acts properly on G/H. 

3) We have dmiH > 2 and, for each i = 1,2, there exists a sequence hi{n) 
hi{n) ^ oo as n ^ cxd, and P2{hi{n)) x ||pi(/ij(n)") \\^' for n G Z"*". 

Proof (0 =^ ID See Proposition ^. 
(0 = 



in H, such that 



This follows from Proposition 3.12, Theorem |1.6|, and Proposition p.3| 



^ IH) ^^ begin by showing that, instead of only sequences /ii(n) and h2{n), there are two 
continuous curves h'^^t) and h'2{t), t G [0, cxd), in H, such that h'^{t) — > cxd as t — s- oo, and, for each 
i = 1,2, we have p2{h'-{t)) x ||pi(^i(0) ll'^S ^^^ * ^ [0,oo). Fix i G {1,2}. For each n, and each 
G > let 

A{n,G) = {a£ fi{H) \ \\a\\ > n and G-^\\pi{a)p < \\p2{a)\\ < G||pi(a)f' }. 

The existence of the sequence {/ij(n)} implies that there is some Gq > 0, such that, for every n, the 
set A{n,Go) is nonempty. Because A{n,Go) is exp-definable (c.f. ^.H]) , Lemma 3.12 implies that 
we may choose an unbounded path-connected component A„ of A{n, Gq), for each n. Furthermore, 
we may assume that An+i C A^. Now, for each natural number n, choose a point a„ G An, and 
let h[{t)n<t<n+i be any path in An from a„ to a„_|_i. 

Since H is homeomorphic to some Euclidean space M™', with m > 2, it is easy to find a continuous 
and proper map $ : [1,2] xR+ ^ H such that ^{i, t) = h[{t) for i = 1,2 and for all t G M+. From 



Proposition 3^ and the definition of ki, we know that the curve /i(/i^(t)) stays within a bounded 
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distance from the wall Lj; say dist [($(i, t)) , Lj] < C for all t. We may assume that C is large enough 
that dist($(s, 1), e) < C for all s G [1,2]. Then an elementary homotopy argument shows that 
/i[$([l,2] X M+)] contains { a G A+ | dist(a,Li U L2) > C}, so /i[$([l,2] x ]R+)] Ri ^+. Because 
fJ,{H) D /i[$([l,2] X M"^)] , we conclude from Theorem |1.6| that H is a Cartan-decomposition 
subgroup. D 



3.15. Remark. When verifying Condition |3.14 (3|) for a specific example, one may use any matrix 



norm, because any two norms are equivalent up to a bounded factor. In practice, the authors use 
the maximum absolute value of the matrix entries, but the reader is free to make another choice, 
because we calculate our results only to within a bounded factor. In our applications to SL(3,M) 
and S0(2,n), we have p2 = pi A pi, the second exterior power of pi. So we calculate ||/02(^)|| as 
the maximum absolute value among the determinants of all the 2x2 submatrices of pi{h). 

If dimi^ = 1 and M-rankG > 2, then we know, from Proposition |3.13 , that H is not a Cartan- 



decomposition subgroup of G. For completeness, the following simple proposition describes n{H) 
fairly explicitly. Namely, p{H) consists of either one or two rays, or one or two logarithmic curves. 

3.16. Proposition. Assume that M-rank(G) > 2, and let H be a nontrivial one-param.eter sub- 
group of AN. 

1) If H is conjugate to a subgroup of A, then there is a ray R in A'^ , such that p{H) ~ R\Ji{R), 
where i: A^ -^ A^ is the opposition involution {see \4.1^ ). 

2) If H C N, then there is a ray R in A'^ , such that p.{H) ~ R. 

3) If neither (ni) nor (0) applies, then there is a ray R in A'^ , a ray R' in A that is perpendicular 
to R, and a positive number k, such that p{H) k, X \J i{X), where X = {rs \ r & R, sG 
R', \\s\\ = (log||r||)^}. 

Proof. Let H = {/i*}. Replacing H hy a conjugate, we may assume that /i* = a*u*, where {a*} 
is a one-parameter subgroup of A and {u*} is a one-parameter subgroup of N, such that {a*} 
centralizes {n*} (c.f. [2.41). The subgroup {u*} is contained in a subgroup S" of G that is locally 



isomorphic to SL(2,M) [|Jac| , Thm. 17(1), p. 100]. Replacing H hy a conjugate, we may assume 
that the intersection As = ^ n S is nontrivial. 

(HI) We have {/i*} = {a*}. Replacing H hy a conjugate under the Weyl group, we may assume 
that a* G A+ for all t > 0. Let i? = { a* | t > }. 

(g) We have {/i*} = {u*}, so H is contained in 5. Then, because both H and As are Cartan- 
decomposition subgroups of S (see [l.Tp , we have p-{H) x p{S) x p{Ag). Let R = p{Ag). 

(^) Both a* and u* are nontrivial. Because {a*} centralizes {u*}, and hence centralizes all of S, 
we know that As is perpendicular to {a*} (with respect to the Killing form). Let /xg: S" — > A^ be 
the Cartan projection of S. (We may assume that K S is a maximal compact subgroup of S.) 



Assuming, for simplicity, that {a* | t > 0} C A~^, we see from the proof of Proposition 3.17 that 
/x(/i*) = a*/is(^*)- Then, because ||a*|| grows exponentially and ||n*|| grows polynomially, the desired 
conclusion follows, with R = {a* | t > 0} and R' = A^. D 

3.17. Proposition. Assume that ]R-rank(G) = 2, and that uj is a real root of G. Let H be a 
closed, connected subgroup of G, such that () = t + u, for some one- dimensional subspace i of a and 
some nontrivial subalgebra u of u^ +U2aj- 

1) Let a^j = [Ua;,u_t^] n a, a one- dimensional subspace of a. 

2) Choose rays t+ in t and a+ in a^^, such that (t | a) > for every t G t^ and every a G a^, 
where (• | •) is the inner product on a defined by the Killing form. 

The subgroup H is a Cartan-decomposition subgroup if and only f/t+ — {0} is not contained in the 
interior of the region C defined as follows: 

a) if a^ is contained in the interior of a Weyl chamber, let C be the interior of that Weyl chamber; 
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b) if a'^ is the wall between two Weyl chambers Ci and C2, let C be the interior of Ci U a^^ U C2. 

Proof. Let M = {U^, U2uj, U-^j, U-2iu), so M is semisimple of real rank one. (The only real roots 
of M are ±aj and, possibly, ziz2uj.) Let A^ = M n ^, so A^^ is a maximal split torus of M, and 
note that the Lie algebra of A^^ is o^^. The choice of the ray a^^ determines a corresponding Weyl 
chamber A^ in A^^. 

We may assume that Km = i^flM is a maximal compact subgroup of M, and let hm '■ M -^ A^ 
be the corresponding Cartan projection. 

Let S = Ca{M). The subgroup MA is reductive, so we have MA = MS. In particular, any 
element of A may be written uniquely in the form as with a G A^^ and s G S. We extend //m to a 
map ^MA '■ MA —^Ahy defining fiMAirns) = ^M{'ni)s for m G M and s G S. 

Because HnN C U^U2uj C M and T C A, we have H C MA. 

Claim. We have ^ma{H) = A'^ ^ma{T). Given h £ H, we may write h = ut, with u £ H (1 N 
and t £ T. Furthermore, we may write t = as, with a G A^^ and s £ S. Choose k, k' £ Km with 
kuak' = iiM{ua) £ A'^. 

Assume for simplicity that G is a matrix group, that is, G C GL(£, M), for some L Then, for 
g £ G, we may let \\g\\'^ he the sum of the squares of the matrix entries of g. Assume, furthermore, 
that K = G n S0(£), that A is the group of diagonal matrices in G, and that A^ is the group of 
unipotent upper-triangular matrices in G. Then it is clear that \\ua\\ > \\a\\ and, because || • || is 
bi-X-invariant, that ||/i(5')|| = \\g\\ for all g £ G. 

Because 

\\fJ.M{ua)\\ = \\ua\\ > \\a\\ = \\iJ,Mia)\\, 
and both of iiM{ua) and fiM{ch) are in A^, there is some a^ £ A^ with ^M{ua) = a'^ ixm[o). 
Therefore 

fJ-MAih) = fiMA{uas) = iiM{ua)s = a'^fiM{a)s = a'^fiMA{as) £ A';^fiMAiT). 

Therefore fiMA{H) C A+ fiMA{T). 

Conversely, given a'^ £ A^ and t £ T, write t = as with a £ A^ and s £ S. Because ||a+/iM(a)|| > 
||MM(a)|| = ||a||, there is some u £ H n N with ||ua|| = ||a''";UM(a)||- Then HM{ua) = a'^^M{a), 
because there is only one element of A'^ with any given norm, so 

IJ-MAiut) = HM{ua)s = a'^fiM{a)s = a^^MA{t). 

This completes the proof of the claim. 

Note that fiMAig) G KMgKM C KgK, so fi{g) = fi{fiMAig)), for ah g £ MA. 

Now, to clarify the situation, let us define /Xm+o- m + ^ a by ;UMA(exp2;) = expi^fim+aiz)) for 
z £ m + a, and let us introduce a convenient coordinate system on the Lie algebra a = M^. Let 
the X-axis be s, the Lie algebra of S, and let the positive y-axis be a+. In these coordinates, the 
restriction of fim+a to is given by Hm+ai^jV) = i^j \y\)- The line t has an equation of the form 
y = ruiX, so fira+a{T) has the equation y = \7nix\. Thus, the claim asserts that 

^im+a{H) = {{x,y) \y> \mix\ }. 

For some constant nic, we have 

C = {{x,y) \y > |mcx|}. 

Thus, t^ — {0} is in the interior of C if and only if |mt| > I'^cl- 

(<^) If t+ — {0} is not in the interior of C, then, because |mt| < \mc\, we see that iJ,m+a{H) 
contains the closure of C. By definition, we know that the closure of C contains a Weyl chamber, 
so we conclude that n{H) D A~^ , so -ff is a Cartan-decomposition subgroup. 

(=>) If t^ — {0} is in the interior of C, then, because |mt| < \itic\, we see that fim+a{H) is a 
proper subcone of C. If C is a single Weyl chamber, then this immediately implies that H is not 
a Cartan-decomposition subgroup. Now assume the other possibility, namely, that the y-axis a^ 
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is the wall between two Weyl chambers Ci and €2- Assume for simplicity that C2 = A~^ . The 
reflection across the y-axis is the Weyl reflection that maps Ci to C2. Because fim+aiH) is invariant 
under this reflection (that is, the inequality defining fixa+aiH) depends only on \x\, not on x itself), 
we see immediately that fJ-{H) = C2 fl Hm+aiH) is a proper subcone of €2- So H is not a Cartan- 
decomposition subgroup. D 

3.18. Corollary (of proof). Let G, uj, H , T , M , and a^ he as in the statement and proof of 
Proposition 3.1% If H is not a Cartan- decomposition subgroup, then the Cartan projection fJ.{H) 



is the image under a Weyl element of either the closed, convex cone hounded by ^ima{T) {if ci^ is 
in the interior of a Weyl chamber) or the closed, convex cone bounded by A^ and a ray ofT {if a'^ 
is a wall of a Weyl chamber) . 

3.19. Example. Suppose that H is of the form ( |1.9D for some p,q S M, not both zero. As an 



illustration of the application of Proposition 3.17, we show that H is a Cartan-decomposition 



subgroup of SL(3,IR) if and only if either p + q < —m.ax{p,q} or p + q > —m.m{p,q}. Let 
O-oj = {diag(a, —a,0) \ a > 0}. Replacing p and q by their negatives results in the same group, so 
we may assume that p > q. Then we may choose t^ = { diag{pt, qt, —{p + q)t \ t > 0}. The Weyl 
chamber 

C = {diag(a, 6, -(a + 6)) | a > -{a + b) > b} 

contains a^ in its interior. It is clear that t"*" is in the interior of C if and only ii p > —{p + q) > q. 
So t^ is a Cartan-decomposition subgroup if and only if this condition fails, that is, if and only 
if either —{p + q) > p or —{p + q) < q- Because we have assumed that p > q, which means that 



p = uiax{p,q} and q = uim{p,q}, this is the desired conclusion. (The condition in Theorem 1.5 
that max{p, g} = 1 and inm{p,q} > —1/2 is obtained by assuming that max{p, g} = 1, instead of 
assuming that p > q, as we have here. Replacing the pair (p, q) with any nonzero scalar multiple 
results in the same subgroup, so assuming that inax{p,q} = 1 results in no loss of generality.) 

The following proposition shows that characterizing the Cartan-decomposition subgroups of a 
reductive group reduces to the problem of characterizing the Cartan-decomposition subgroups of its 
almost simple factors. Thus, our standing assumption that G is almost simple is not as restrictive 
as it might seem. 

3.20. Proposition. Suppose that G is a connected, reductive, linear Lie group, let G = KAN be 
an Iwasawa decomposition of G, and write G = G1G2 • • • GmZ , where Z is the center of G, and 
each Gi is a connected, almost simple, normal subgroup of G. Let H be a closed, connected subgroup 
of AN. 

1) H is a Cartan-decomposition subgroup of G if and only if (a) H contains Z f] A, and (b) for 
each i, the intersection H OGi is a Cartan-decomposition subgroup of Gi. 

2) No closed, noncompact subgroup of G acts properly on G/H if and only if (a) H contains ZnA, 
and (b) for each i, no closed, noncompact subgroup of Gi acts properly on Gi/{H DGi). 

Proof. We prove only the nontrivial direction of each conclusion. 

(|l|) Suppose that if is a Cartan-decomposition subgroup of G. (a) The proof of the claim in 
Proposition 3.17 , with G1G2 ■ ■ -Gm in the role of M, shows that if contains Z D A. (b) From 
Theorem L6, we know that there is a compact set C C A, such that, for each a G Aj , there is some 
h{a) € H, such that fi{h{a)) G aC. The Cartan projection of G/Gi is a proper map, and, because 
HGi is closed (see [[Hoc| , Thm. XII. 2. 2, p. 137]), the natural map H/{H n Gi) — > G/Gi is a proper 
map. Hence, there is a compact subset Ci of H, such that h{A'\^) C Ci(iinGi). Then Corollary ^ 
implies that there is a compact subset C2 of Ai, such that ^{H n Gi)G2 D ^(/i(A^)) ^^ A'l . 
Therefore, if n Gi is a Cartan-decomposition subgroup of Gi. 
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(^) Proof by contradiction. Assume that (a) and (b) hold, and let L be a closed, noncompact 
subgroup of G that acts properly on G/H. By passing to a subgroup, we may assume that L is 



cyclic. Then there is a one-parameter subgroup L' of G, such that L' ~ L, and, from Lemma 2.9, we 
may assume that L' C AN . From (a), we see that we may assume that L' C Gi ■ ■ ■ Gm, so we may 
write L' = {/*} and /* = /*••• /^, where {Ij} is a one-parameter subgroup of Gj. Every nontrivial, 
connected subgroup of AN is closed and noncompact, so, for each i, (b) implies that either {/*} is 
trivial or {Ij} does not act properly on Gi/{H n Gi). Then, from Corollary 3.5, we see that, for 



each i, there is a compact subset Gi of vl n Gj, such that { t > j /* € ^i{H n Gi)Ci } is unbounded. 
(By taking Gi to be a large ball, we may assume that Gi is exp-definable.) Because this set is 
exp-definable (c.f. 3.1l| ), Lemma |3.12| implies that there is some T > 0, such that, for each i and 



for all t > T, we have /* e ^Xi{H n Gi)Gi. Therefore, for all t > T, we have /* G fi{H)GiG2 • • • G, 



mj 



SO L' = {/*} does not act properly on G/H. This is a contradiction. D 

3.21. Lemma. Let H be a closed, connected subgroup of G. The subgroup H is a Cartan- 
decomposition subgroup of G if and only if H is a Cartan- decomposition subgroup of the identity 
component of G. 

Proof. Let G° be the identity component of G. Because every element of the Weyl group of G 



has a representative in G° [BT, Cor. 14.6], we see that G and G° have the same positive Weyl 
chamber A'^ , and the Cartan projection G° — > A'^ is the restriction of the Cartan projection 
G — > A'^ . Thus, the desired conclusion is immediate from Theorem 11.61. D 



4. CarTAN-DECOMPOSITION SUBGROUPS OF SL(3, 



In this section, we find all the Cartan-decomposition subgroups of SL(3,R) (see 4.5). With 



Remark 2.1C in mind, we restrict our attention to finding the Cartan-decomposition subgroups 



contained in AN . 

4.1. Notation. To provide a convenient way to refer to specific elements of AN, we define 

h: {M+f X M3 ^ GL(3,M) by 

/i(ai,a2,a3,'Ui,'U2,'"3) = 



Ol 


Ul 


^3 





02 


U2 








as 



4.2. Notation. Let 

(4.3) A = {a = h{ai, 02, 03, 0, 0, 0) | Oj G M"*", 010203 = 1 }, 

A~^ = { /i(oi, 02, 03, 0, 0, 0) I Ol > 02 > 03 > 0, 010203 = 1 }, 
N = { /i(l, 1,1, u,v,w) \ u,v,w £M.}, 
and K = S0(3). 

4.4. Notation. We let a and f3 be the simple roots of SL(3,M), defined by a{a) = 01/02 and 
/3(o) = 02/03, for an element o of ^ of the form (|4.3|) . Thus, 

• the root space Uq, consists of the matrices in which all entries except hi^2 a^^e 0; 

• the root space u^ consists of the matrices in which all entries except /i2,3 are 0; and 

• the root space Ua+/3 consists of the matrices in which all entries except /ii^3 are 0. 

4.5. Theorem. Assume that G = SL(3,M). Let H be a closed, connected subgroup of AN, and 
assume that H is compatible with A (see \2. ^ ) . 

The subgroup H is a Cartan-decomposition subgroup of G if and only if either: 
1) dim// > 3; or 
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2) H = A; or 

3) for some p ^ 0, we have H 

4) for some p ^ 0, we have H 

5) for some p ^ 0, we have H = { /i(e~^*, e*,e*, s, 0,ps) | s, t € M}; or 

6) H = T \K U^, for some one-parameter subgroup T of A and some positive root oo, and H 
satisfies the conditions of Proposition 3.1% 



{/i(l,l,l,s,ps,t) I s,t G M} C iV; or 
{/i(e*,e*,e"2*,0,s,ps) | s,t G M}; or 
{ /i(e~^*,e*,e*,s,0,ps) 



4.6. Remark. If p = in 4.5(^ or [4.5| (|q), then H is a. Cartan-decomposition subgroup, but H is of 
the type considered in 4^@. However, if p = in ^]^(y), then H is not a Cartan-decomposition 
subgroup. 



Theorem 4.5 describes the Cartan-decomposition subgroups of SL(3, ] 
subgroups that are not Cartan-decomposition subgroups. 



We now describe the 



4.7. Corollary. Assume that G = SL(3,R). Let H be a closed, connected subgroup of AN , and 
assume that H is compatible with A {see \2. 4 ) . 

The subgroup H fails to be a Cartan-decomposition subgroup of G if and only if either: 

1) dim// < 1; or 

2) f) = Uq + Ua+p or\)=\ip + Ua+p; or 

3) f) is of the form f) = (ker(a — /?)) + u, for some one- dimensional subspace u o/Uq + up, with 
u i {Ua,up}; or 

4) dimF = 2 and H ^ {H D A) \x {H D N); or 

5) H = T t< U^, for some one-parameter subgroup T of A and some positive root to, and Propo- 



sition p. j?| implies that H is not a Cartan-decomposition subgroup. 



In the course of the proof of Theorem 4.5, we calculate the Cartan projection of each subgroup 



that is not a Cartan-decomposition subgroup. Proposition 4.12 collects these results (see also Corol 



lary 3.15 and Proposition 3.16). The statement of this proposition and the proof of Theorem 4.5 
are based on Proposition 3.14, so we describe the required representations pi and p2 of G. 



4.8. Notation. Define a representation 

p : SL(3, M) -^ SL(m3 A M^) by p{g) = g A g. 
In the notation of Proposition |3.3| , we have 

pi = the standard representation on M'^ {Piio) = 9)^ 

P2 = P {=piApi), 
ki = 1/2 and k2 = 2. 

Thus, Proposition |3.3| yields the following fundamental lemma. 

4.9. Lemma. Assume that G = SL(3,M). A subgroup H of G is a Cartan-decomposition subgroup 
if and only if p{H) ~ [||/i||-'^'^, ||/i|p], where the representation p is defined in Notation ^7|. 

4.10. Notation. Let i be the opposition involution in yl+, that is, for a € A'^ , i{a) is the unique 
element of A'^ that is conjugate to a~^, and set B'^ = {a £ A~^ \ i{a) = a }. 

4.11. Corollary. Assume that G = SL(3, M). Let H be a closed, connected subgroup of G with 
diva. H — diva. K H > 2, where Kh is a maximal compact subgroup of H . The subgroup H is a Cartan- 
decomposition subgroup if and only if there is a sequence /i^ — > oo in H with p{hn) ^ ||^n|P- 



Proof. (<^) We may assume that H C AN (see 2^). If we identify the Lie algebra o of j4 with 
the connected component of A containing e, then A"*" is a convex cone in a and the opposition 
involution i is the reflection in A'^ across the ray B^ . If Li and L2 are the two walls of the Weyl 
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chamber A^ , then i{Li) = L2. Therefore, because fJ.{hn) is a bounded distance from one of the 
walls, we know that i[fi{hn)) is a bounded distance from the other wall. That is, p[i(/x(/i„))] x 

||i(/i(^„)) II . In other words, we have p{h~^) x ||/i^^|n^- Therefore, using the sequences /i„ 
and h~^, we conclude from Proposition p.l4| that i:f is a Cartan-decomposition subgroup. D 



We now describe the Cartan projections of the subgroups that are not Cartan-decomposition 
subgroups. 

4.12. Proposition. Assume that G = SL(3,M). 

), then p{h) x h for every h (z H . 
(j), then p{h) x h for every h a H. 



1) If H is of type 4- 

2) // H is of type 0( 

3) IfH IS oftype^lg), then p{H) x [{\\h\\log\\h\\)^/'',\\hr /{log\\h\ 



Proof of Theorem |4.5| . If A C H, then H is a Cartan-decomposition subgroup (because A is 
a Cartan-decomposition subgroup) and we have either dimH > 3 or H = A. Thus, we may 



henceforth assume that A (^ H . Then, from the proof of Lemma 2.8, we see that A is not contained 



in the Zariski closure H of H. Therefore dim(^ n H) < 1, so there are a, 6, c G M with 
(4.13) Hr\A = { h{e°-\ e^\ e"*, 0, 0, 0) | t G M }. 

and a + 6 + c = 0. (li Ar\ H = e, then a = 6 = c = 0.) Because a 1-dimensional subgroup cannot 
be a Cartan-decomposition subgroup (see 3.13), we may assume that dmiH > 2. 



Case 1. Assume that H <Z N and that dimi/ = 2. We show that H \s a, Cartan-decomposition 
subgroup if and only if H is not normalized by A. (That is, if and only if H is of type (^.) In the 
case where H is not a Cartan-decomposition subgroup, we show that p{h) x h for every h a H. 

Every 2-dimensional subalgebra of n contains Ua+f3, so we must have u^^^ C f|. 

(=>) We prove the contrapositive. Thus, we suppose that H is normalized by A, so f) is a sum 
of root spaces. Therefore, f) is either Uq + Ua+p or u^ + Ua+p- In either case, every element of H is 
conjugate, via K, to an element of Ua+13, so p{H) = ^{Ua+f^)- But Ua+fj, being one-dimensional, 
cannot be a Cartan-decomposition subgroup (see |3.13| ), so we conclude that H is not a Cartan- 
decomposition subgroup. 

{^) Because H is not normalized by A, and contains U0+/3, we must have H = {h £ N \ hi^2 = 
P^2,3 }, for some nonzero p G M. For each t G M, let ht = /i(l, 1, l,pt, t, 0). Then p{ht) x t^ x ||/it|p, 
so if is a Cartan-decomposition subgroup. 

Case 2. Assum,e that dim if > 3. We show that ii is a Cartan-decomposition subgroup. 

If dim if > 4, then H must contain either ^ or A^ (c.f. |2.8| ), so ii is a Cartan-decomposition 



subgroup. Thus, we may assume that dim if = 3. Furthermore, from Lemma 2^, we may assume 
that dim(ii n iV) = 2. 

We may assume that if n A^ is normalized by A^ for, otherwise. Case || implies that H nN (and, 
hence, H) is a Cartan-decomposition subgroup. So f) n n must be either u^ + Ua+/3 or U/3 + Ua+/3- 
For definiteness, let us assume that f) n n = Uq + Ua+/3- (The calculations are essentially the same 
in the other case. In fact, u^ + Ua+is and U/3 + Uq+^j are conjugate under an outer automorphism 

Offl.) 

Subcase 2.1. Assume that H = {H D A) t< {H D N) . We may assume that a < 0. (Recah that 
a,b,c are defined in ( [4.131) .) For each t G M+, define /i* = /i(e"*,e''*, e'^*,e'^*,0, e*^*) G H, where 
d = max{6, c} > 0. Then p(/i*) x e^'^* x |[/i*|p, so if is a Cartan-decomposition subgroup. 

Subcase 2.2. Assume that H / (H n ^) x {H n N). Let w be a positive root as described in 



Lemma 2A. Since Ua,Uo+/3 C i), we must have uj = p. Thus, for some nonzero p, we have 

if = {h{e\e\e~'^\u,pte\v) \t,u,v eR}. 
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For each t £ M+, let h = h{e\ e\ e'^^ pte\ pte\ 0) G H. Then p{ht) x t^e^* x \\ht\\^, so i:f is a 
Cartan-deconiposition subgroup. 

Case 3. Assume that dmiH = 2, that H ^ N, and that H = {H n A) ix {H n N). We may assume 



that H n N is not a root group, for otherwise Proposition 3.17 apphes. We show that H is a 
Cartan-decomposition subgroup if and only ii H f] A = kera oi H f] A = ker/3. (That is, if and 
only if H is of type (Q) or (^), respectively.) When H is not a Cartan-decomposition subgroup, we 
have H A = ker(a — /?), in which case we show that p{h) x h for every h £ H. 

Because H (1 N is normalized by H A, and is one-dimensional, every element of f} fl n is an 
eigenvector for each element oi H (1 A. Because, by assumption, t) n n is not a root space, this 
implies that two positive roots must agree on H A. So i? n ^ is the kernel of q, /?, or a — /?. 

li H n A = kera ox H d A = ker/3, then we have p{a) x ||a|p or p{a~^) x ||a~^|p, for every 
a G H f] A. Thus, H is a Cartan-decomposition subgroup. 

Assume now that H D A = ker(a — /?). Then 

H = {h{e\l,e~\e^u,pu,e^pu^ /2) \t,ueR}, 

for some nonzero p G M. Thus, we have p{h) x max{e*n^, e^*} x h for every h G H with |t|, \u\ > 1. 
Any element of H is within a bounded distance of some point with |tl, |n| > 1, so we conclude from 



Lemma 3.4 that p{h) x h for every h G H. Therefore, H is not a Cartan-decomposition subgroup. 

Case 4- Assume that diuiH = 2 and that H ^ {H D A) \x {H D N) . We show that H is not a 
Cartan-decomposition subgroup, and that 

piH)^[\\h\\'/\log\\h\\),\\hf/ilog\\h\\)]. 



Let the positive root uj and the subgroup U of N he as described in Lemma 2.4. Assume for 
definiteness that oj = a. (The calculations are similar in the other cases. Indeed, the groups in the 
other cases are conjugate to these under an automorphism of G.) Because Ua is one-dimensional, 
we must have -(/'(kera) = Ua- 

Because the restrictions of /? and a + /? to kera are nontrivial, unlike the restriction of a, 
and kera normalizes U, but Ua 't- U, we see that U C UjsUa+fs- Then, because U = H n N 
is one-dimensional and is normalized by Ua, we conclude that H (1 N = Ua+p- Thus, we have 
H = {/i(e*, e*,e~^*,pte*,0, s) | s,t G M}, for some nonzero p G M. For \s\,\t\ > 1, we have 
h X max{|t|e*, e~^*, \s\} and p x max{e^*, |i|e~*, |s|e*}. 

Letting s = te* » yields p{h) x te^* x ||/i|p/(log ||/i||). We now show that this is (ap- 
proximately) the largest possible size of p{h) relative to h. Because p{H) is invariant under the 
opposition involution (see 4.101) , this implies that p{H) x [(||/i|| log ||/i||)^'^, ||/i|p/(log |j/i||)] , so H 
is not a Cartan-decomposition subgroup. For t < —1, we have e^* < 1, |i|e~* < e~^*, and |s|e* < |s|, 
so ||/o(/i)|| = 0(||/i||) is much smaller than ||/ip/(log ||/i||). Now suppose that t > 1. If |s[ < [t[e*, 
then 

p{h) = 0{te^') = 0[{te'f/t) = 0{\\hf/\oi 

If \s\ > |t|e*, then 



p{h) = 0{\s\e') = 0\---r\=0\-—\=0 



kl/e*/ \logsy \log|I/i| 

as desired. D 



Proof of Theorem 1.8. Let H he a minimal Cartan-decomposition subgroup of G = SL(3,] 



Case 1. Assume that H C AN and that dimH < 3. Consider the possibilities given by Theorem [4.5| . 
If H is of type 45(|2|), then H = A is listed in Theorem L8. If H is of type [4.5| (|^), then we may 



assume that p = 1, by replacing H with a conjugate via an element of A] thus, H is the subgroup 
of A^ that is listed in Theorem 1^. li H is oi type 45|(^ or 45 (5|), then we may assume that 
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p = 0, by replacing H with a conjugate via an element of S0(2) x Id or Id x S0(2), respectively (a 

maximal compact subgroup of the centralizer oi H r\ A); thus, H is of type |4.5| (|^), discussed below. 

If H is of type ^]^(P), then, because all roots are conjugate under the Weyl group, we may 

assume that uj = a] thus, H is of the form (1.9). Example 3.18| shows that we may assume that 



max{|?, q} = 1 and that min{p, g} > —1/2, so H is listed in Theorem 1.5 



Case 2. Assume that H C AN and that dimff > 3. Because H is minimal, we know that H 
does not contain any conjugate of A, so d\m.{H n N) > 2 (see ^ ). Then, because H (1 N is not 
a Cartan-decomposition subgroup, we see from Corollary |4.7| that dim{H n N) = 2 and that \r) is 
either u^ + Ua+/3 or u^ + Ua+p- There is an automorphism of G that normalizes A and Ua+p, 
but interchanges Ua with C/g (namely, (f) is the composition of the Cartan involution ( "transpose- 
inverse" ) with the Weyl reflection corresponding to the root a + (3), so we may assume that H = 

li H n A is nontrivial, then, because the Weyl chamber containing At and the Weyl chamber 
containing ^^.a have no common interior, we see from Proposition 3.17] that either {H Pi A) k Ua 



or [H n A) tK Ua+j3 is a Cartan-decomposition subgroup, which contradicts the minimality of H . 



Thus, Hr\A must be trivial, and we may assume that H is compatible with A (see 2.3), so, from 



Lemma 2.4, we see that H is conjugate (via an element of A) to the three-dimensional subgroup 
of AN that is listed in Theorem |1.8| . (We must have u; = a, because H contains Up and U^+p-) 



Case 3. Assume that H is not conjugate to a subgroup of AN . Any nontrivial, connected, semisimple 
Lie group has a connected, proper, cocompact subgroup (namely, either the trivial subgroup or 
a parabolic subgroup, depending on whether H is compact or not), so the minimality implies 



that H is solvable. From the nilshadow construction (cf. proof of Lemma 2.£), we may assume, 
after replacing H hy a conjugate subgroup, that there is a connected subgroup H' of AN and 
a compact, connected, abelian subgroup T of G, such that T normalizes both H and H' , and 



we have HT = H'T. We may assume that H' is compatible with A (see 2.3). Because H' Ci N 



is a nontrivial, connected subgroup of N that is normalized by a nontrivial, connected, compact 
subgroup of G, we must have either H' n N = UaUa+p or H' (1 N = UpUa+p- Replacing H hy a 
conjugate via an automorphism of G (cf. the automorphism (p described in Case |2|), we may assume 
that H' n N = UpUa+p- Then H' (1 N is not a Cartan-decomposition subgroup, so H' ^ N. Let 



Th = (HN) n A. From Lemma 2A, we know that Th is contained in the Zariski closure of H' , so, 
by replacing H with a conjugate, we may assume that T centralizes Th- Thus, Th = kero;, for 
some root to, and T C {U^j, U-cu)- Because UpUa+p = H' Ci N is normalized by T, we must have 
uj = a. 

Now TTh is a maximal connected subgroup of the centralizer Cg{Th) (because S0(2) is maximal 
in SL(2,M)), so 

H' n Gg{Th) = {h'tTh) n Gg{Th) nH' = {tTh) nH' c Th. 



Thus, H' is not of type |j(|), so Lemma U imphes that H' = {H' n A) x {H' n N). Therefore, 
H C (Tkera) x (UpUa+p), so, after conjugating T to a subgroup of S0(2) x Id, it is easy to see 
that H is of the form ([TlOl ). D 

5. Subgroups of S0(2,n) contained in A^ 
We now study G = S0(2,n). In this section, we determine which subgroups of N are Cartan- 



decomposition subgroups of G. Theorem p.3| gives simple conditions to check whether a subgroup 
of A^ is a Cartan-decomposition subgroup, but the statement of the result requires some nota- 
tion. Theorem |5.5| d escribes all the subgroups of A" that are not Cartan-decomposition subgroups, 
and Proposition ^.g| describes the image of each of these subgroups under the Cartan projection. 
Analogous results for subgroups not contained in A^ appear in Section 0. 



18 



HEE OH AND DAVE WITTE 



5.1. Notation. Recall that we realize S0(2,n) as isoinetries of the indefinite form {v \ v) = 
viVn+2 + V2Vn+i + X^ILs ^f °^ M'^"'"^, that A consists of the group of diagonal matrices, and that N 
consists of the upper-triangular unipotent matrices (see 1.12D . Let 



A^ = {diag(ai,a2,l,l,... ,1,1,03^ 



, U,;^ 



ai > 02 > 1 } 



and K = S0(2, n) n SO(n + 2). 



5.2. Notation. Given (j),r] ^^ and x, y G M" (and letting ti = ^2 = 0), there is a corresponding 
element of n (see 1.13 ). The exponential of this is an element h = h{(j),x,y,r]) of A^. Namely, 



/l (p x + ^4>y ri-^{x-y)-U 



2 

y 

Id 



2II4/11 

T 

-y 
1 



|y|| 



-V- 



M\xP + ^ 

9 K' T^ 0/1 



Ux-y) + l 



2 II Ii2\ 



+ Uy 



¥ 



\ 



J 



Because the exponential map is a diffeomorphism from n onto N, each element of N has a unique 
representation in this form. Thus, each element of N determines corresponding values of (p, x, y, 
and rj. We sometimes write (ph, Xh, yhi and rjfi for these values, to emphasize the element h of N 
that is under consideration. 

The following description of the Cartan-decomposition subgroups contained in A*" is obtained by 
combining Lemmas 5.11 and ^.14 with Proposition 3.13| . 



5.3. Theorem. Assume that G = S0(2,n). A closed, connected subgroup H of N is a Cartan- 
decomposition subgroup of G if and only if either 

1) dim if = 2, Uq+2/3 C f), and there is an element of t) such that (j) ^ and y 7^ 0; or 

2) dimH > 2 and 

(a) there is an element of i) such that {{(j),x), (0, y)) is one- dimensional in W^~^ ; and 

(b) there is a nonzero element of f) such that either 



(i) ((0, j;), (0, y)) is two-dimensional in 
(ii) y = and \\x\\^ = —2cl)r]. 



on— 1 . 



or 



5.4. Notation. We let a and (3 be the simple real roots of S0(2,n), defined by a{a) = ai/a2 and 
/3(a) = 02, for an element a of A of the form 



a = diag(ai,a2, 1,1, 
Thus, 

• the root space Uq, is the cp-axis in n, 

• the root space up is the y-subspace in n, 

• the root space Ua+p is the x-subspace in n, and 

• the root space Ua+2f3 is the r/-axis in n. 



,l,l,a2\«i^) 



Theorem 5.3 describes Cartan-decomposition subgroups. The following result describes the sub- 
groups of N that are not Cartan-decomposition subgroups. It is obtained by combining Corollaries 
5.13 and [5.15 with Proposition p.l3| . 



5.5. Theorem. Assum,e that G = S0(2,n). A closed, connected subgroup H of N is not a Cartan- 
decomposition subgroup of G if and only if either 

1) dimH < 1; or 

2) for every nonzero element of\), we have </) = and dim(j;, y) 7^ 1 {i.e., dim(x,y) € {0,2}); or 

3) for every nonzero element of f), we have (/> = and dim(x,y) = 1; or 
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4) there exists a subspace Xq of . 



b € Xq, c € Xq , and p € M with \\b\\ 



2p < 0, 



pcf) + b ■ X {where b ■ x 
In the course of the proof of Theorem |5.5|, we calculate the Cartan projection of each subgroup 



such that for every element of I), we have y = 0, x ^ (l)c + Xq, and rj 
denotes the Euclidean dot product of the vectors b and x in M""^). 



that is not a Cartan-decomposition subgroup. Proposition 5.8 collects these results. The statement 



(and all of the proofs in this section) is based on Proposition 3.14, so we describe the required 
representations pi and p2 of G. 



a: 



by p{g) = g /\g- 



pi 

P2 

ki 



(piig) =g), 



5.6. Notation. Define a representation 

p: S0(2,n) ^ SL(M"+2 
In the notation of Proposition |3.3| , we have 

the standard representation on 
p (=piApi), 
1 and ^2 = 2. 

yields the following fundamental lemma. 

5.7. Lemma. Assume that G = S0(2,n). A subgroup H of G is a Cartan-decomposition subgroup 
if and only if p{H) ~ [\\h\\, |I/i|P], where the representation p is defined in Notation 5.6 . 



Thus, Proposition 



5.8. Proposition. Assume that G = S0(2,n) and let H be a closed, connected subgroup of N. 
1) IfH is of type \57^\^) {i.e., ifdimH < 1), then either p{h) x hforallh E H, or p{h) x ||/i|p/2 



for all h G H, or p{h) 

2) // H is of type |5~ ~ 

3) IfH is of type 5.<^(|j| 

4) If H is of type 5. . 



2 for all h£ H. 
id), then p{h) x ||/i|p for all h e H. 



), then p{h) 
{J), then p{h) 



h for all h ^ H . 
h for all h £ H . 



Note that if either p{h) x h for all h G H, or p{h) x ||/ip for all h & H, then p{II) is within a 
bounded distance of one of the walls of A'^. On the other hand, if ii^ is a Cartan-decomposition 
subgroup, then p{II) ~ A~^ . Thus, certain one-dimensional subgroups (described in Lemma 5.20| ) 
are the only connected subgroups of A^ for which p{H) is neither a wall of the Weyl chamber nor 
all of j4+ (up to bounded distance). In Section ^, where H is not assumed to be contained in A^, 
we will see several more examples of this. 



Proof of Theorem 1.14. It is easy to see from Theorem 5.3 that the subgroup of G corresponding 



to each of the subalgebras listed in Theorem 1.14 is a Cartan-decomposition subgroup. Each is 



minimal, because each is 2-dimensional (see 3.13). Lemma 5.1(]| shows that no two of the listed 
subalgebras are conjugate. 

Thus, given a minimal Cartan-decomposition subgroup of G that is contained in A^, all that 
remains is to show that the Lie algebra of H is conjugate to one on the list. 

The Weyl reflection corresponding to the root /3 fixes a + /3, but interchanges a and a + 2/3. 
Therefore, the subalgebra 



(5.9) 



^0 a; Or? 
-r/ 



x,r/ e 



0. 



is conjugate to the subalgebra of type 1.14 (21) with €2 

Case 1. Assume that \) C Ui3+Ua+j3 + Ua+2(3- From Theorem 5^, we know that there is an element u 



of f) with (Mni{xu,yu) = 1- By replacing H with a conjugate under {Ua, U-a), we may assume that 
yu = 0. Then, by replacing H with a conjugate under U/3, we may assume that ??„ = 0. If 
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Ua+2/3 C i), then f) contains a conjugate of the subalgebr a (|5.9|). T hus, we may now assume that 
Uq,_|_2/3 ^ i}- Therefore, i) has no nonzero elements of type 5.3| ( |2(b)i]| ), so there must be some v G H 



with dira{xv,yv) = 2. Because Uq,+2/3 ^ f), we know that i) is abehan, so v commutes with u, which 
means that Xu is perpendicular to yy. Thus, {u,v) is conjugate to one of the two subalgebras of 
type[ri4|@. 



Case 2. Assume that there is an element u of i), such that u ^u^ + Uq+/3 + Uq,+2/3- 

Subcase 2.1. Assume that there exists w € (f) fl {up + Uq,_(_^ + Uq,+2/3)) \ {v^a+i3 + ^a+2i3)- We have 
[u,v\ G (Uq+^ + u„+2/3) \Uq+2/3 and ['u,t;,';;] G Ua+2/3 \ {0}, so {[u,v],[u,v,v]) is conjugate to the 
subalgebra (p^). 



Subcase 2.2. Assume that f) C Uq + Uq+^ + Uo+2/3- No element of f) is of type 5!3| ( |2(b)i ), so there 



must be a nonzero element v oi\] with \\xy\\ = —2(j)yriy. Replacing H hy a conjugate under Uf3, we 
may assume that Xy = 0; then either (py or rjy must also be 0. (Even after conjugation, we must 
have p[exp{tv)) x || exp(tf)|p.) 

If (/)y = 0, then Ua+2/3 C i), so i) contains a conjugate of either the subalgebra ( |5.9| ) or the 



subalgebra 1.14K|1|). 



If r]y = 0, then Uq, C f}. Let w; be a nonzero element of f) n {Ua+/3 + Ua+2i3)- If ^«; = 0, then {v, w) 



is conjugate to the subalgebra of type 1.14(4) with €2 = 0. If r/2 y^ 0, then, by replacing i) with a 



conjugate under C/_/3, we may assume that x^ = 0, so {v,w) is the subalgebra 1.14| ([I] 



Subcase 2.3. The general case. If H is of type |5.3| ([l|), then fj is conjugate to the subalgebra of 
type 1.14| (p]) with ei = 1. Thus, we henceforth assume that H is of type 5.3(2[). 



Let V be an element of [} with dmi{{(j)y,Xy), {0,yy)) = 1. We may assume that Subcase 2^ does 
not apply, so we may assume that yu ^^■ 

Suppose that 4>y / 0. Then yy must be 0. Because (/>^, 7^ 0, y^, = and y^ 7^ 0, there is some 



linear combination of v and u, such that 1;^ = and y 7^ 0, so Subcase 2A_ applies. 

We may now assume that (/>„ = 0. We may assume that y^, is also 0, for, otherwise, Subcase 2.1 
applies. Thus, v € {Ua+p +"^0+20) \'^a+2ii- We may assume that [u,v\ = 0; otherwise, (v, [u,v\) is 
conjugate to the subalgebra (|5.9[) , because [u,v\ G Uq+2/3- 

Conjugating by an element of Up, we may assume that Xu is a scalar multiple of Xy, and that 
r]y = 0. Then, by replacing u with u + Xv, for an appropriate scalar A, we may assume that x^ = 0. 
Recall that yu 7^ 0. Therefore, conjugating by an element of C/q,+/3, we may assume that tJu = 0. 
Because [u,v] = 0, we know that Xy is perpendicular to y„. Thus, {u,v) is conjugate to one of the 



two subalgebras of type 1.14| (^) . D 



5.10. Lemma. Assume that G = S0(2, n). No two of the subalgebras of n listed in Theorem I.Ij^ 
are conjugate under G. {In fact, they are not even conjugate under GL(n + 2,]R). 

Proof. For each subalgebra f), we look at the restriction of the exponential map to f). This is a 
polynomial function from f) into A^. For convenience, we use (</>, ^), ix,y), or ((/>, x) as coordinates 
on f), and use the matrix entries as coordinates on N. 

The subalgebra of type 1.14| (1|) with ei = 1 and the subalgebra of type |1.14| (P) with e2 = 1 are 



the only cases where the exponential has a 4th degree term (namely, 0^/24), so they cannot be 
conjugate to any of the others. They are not conjugate to each other, because in the subalgebra 



of type |1.14| (|lD with ei = 1, there is a nontrivial subspace u such that ;u(expu) ^ \\\h\\ ,\\h 



(namely, u = Ua+213), but there is no such subspace in the subalgebra of type |1.14| (|2D with £2 = 1. 
Of the others, it is only for the two subalgebras of type |1.14| (^) that there are two independent 
quadratic terms (so the polynomial p(exp(ii)) is of degree 4). For the subalgebra with es = 0, one 
or the other of the quadratic terms vanishes if either x or y is (that is, on two lines); whereas, for 
£3 = 1, neither quadratic term vanishes unless y = (a single line). 
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For each of two remaining subalgebras, the exponential function has a single quadratic term. 



For the subalgebra of type 1.14(2|) with e2 = 0, the quadratic term does not vanish unless x = (a 



single line). Whereas, for the subalgebra of type I.14| ([l|) with ei = 0, the quadratic term vanishes 



if either i;^ = or 77 = (two lines) . D 



The remainder of this section has three parts. In §5A|, we determine when /i(-ff) contains elements 



close to the wall given by p{h) x h. In §5B, we determine when n{H) contains elements close to 
the other wall, given by p{h) x ||/i|p. In §5C, we calculate fJ.{H) for some subgroups that are not 
Cartan-decomposition subgroups. 

5A. When is the size of p{h) linear? 

5.11. Lemma. Assume that G = S0(2,n). Suppose that H is a closed, connected subgroup of N , 
and that dim H ^ 1. Then the following are equivalent: 

1) There is a sequence {hn} of elements of H with hn -^ 00, such that p{hn) x /i„. 

2) There is a continuous curve {ht}t<^^+ of elements of H with ht -^ 00, such that p{ht) x ht- 

3) Either 

(a) there is an element of i) such that (((/>, x), (0, y)) is one- dimensional in M""^; or 

(b) dimH = 2, Ua+2^ C i), and there is an element of f) such that 7^ and y 7^ 0. 

We prove the following more general (but slightly more complicated) version that does not assume 
that dimH ^ 1. 

5.12. Lemma. Assume that G = S0(2,n), and let H be a closed, connected subgroup of N . Then 
the following are equivalent: 

1) There is a sequence {km} of elements of H with hm — > 00, such that p{hm) x hm- 

2) There is a continuous curve {ht}t^^+ of elements of H with ht — > 00, such that p{ht) x /if. 

3) Either 

(a) There is an element oft) such that ((0, x), (0, y)) is one- dimensional in M"^-*^, and either 
y 7^ or ||x|p 7^ —24>rj; or 

(b) dimi7 = 2, Uq+2/3 C \), and there is an element of f) such that 4> ^ and y 7^ 0. 

Proof (0 ^ D 



Case 1. Assume that (j) = 0{1) in km- We may assume that (j) = (see 3^). Let am = k'm^mkm 
be the Cartan projection of km, with k'^^km S K, and write a^ = diag(am , • • • ,am ), with 
Hm > a-rn > ' ' ' > Om (bccausc Qm G ^^)- Let 

V = {ve M"+2 \\fi<n, Vi = 0} andW = {we M"+^ \wi = 0}, 

so dim V = 2 and dim W = n + 1. Then dim V + diui{kmW) = n + 3>n + 2, so there is a nonzero 
vector Vm ^ V n {kmW); we may assume that ||fm.|| = 1- 

Because Vm € V, we may write Vm = (0,0, ■ ■ ■ , 0, qm,Pm), so 

hmVm ^ (*i *) ~\PmXm + QmymJ^ *) *) ■ 

Because p{am) x p{hm) ^ hm ^ Omi we must have am = 0(1) for i > 2, so amW = 0{w) for all 
w £ W. Then, since k^Vm G VF, we have 

WPm^m ~T~ QmymW S ||'lmf^m|l — III""™/ 0'm"'m ^m|| 

= \\amkm^Vm\\ = 0{kmVm) = 0{Vm) = 0(1). 

Passing to a subsequence, we may assume that {vm} converges, so {pm} and {qm} converge. Clearly, 
lla^mll + WUmW tends to infinity, because p{h) x \\h\\^ if (/>, x, and y are all zero (so the same is true if 
4),x,y = 0(1) (see U)). Thus, letting x^ = Xm/{\\xm\\ + hmW) and y^ = ym/{\\xm\\ + ||ym|I), we 
have Pmx'm + QmUm = ci(l). Because ||x'„||, \\y'm\\ < 1, we may pass to a convergent subsequence. 
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Then Poox'^ + QooH'oo — 0) so x'^ and y'^ are linearly dependent. Because f) is a closed subset of n, 



there is some element of f) with = 0, x = x^, and y = y'^, so conclusion ( |3a| ) holds. 

Case ^. AssuTTie that (f) is unbounded in {hm}- By passing to a subsequence, we may assume that 

\4>m\ -^ OO. 

Assume for the moment that some element u of f) n (u^ ©Ua+/3 ©Uq_(_2/3) does not belong to Ua+2/3- 
If u ^ Ua+f3 © Uq,_|_2/3 , then [u, log h] belongs to Ua+p © Ua+2i3 1 but is not in Uq,+2/3 • Thus, by replacing 
u with [u, log/i] if necessary, we may assume, without loss of generality, that u G Ua+f3 © Ua+2p- 
This implies that cpu = 0, Vu = and x^ 7^ 0, so dim((0u,x„), (0,yu)) = 1, as desired. 

We may henceforth assume that every element of f) n {up ©Ua+/3 ©u„+2/3) belongs to Ua+2f3- This 
means that x and y are uniquely determined by 0, so dimf/' < 2, and there are (j) S Uq,, x € Ua+p, 
and y € u^, such that (p ^ 0, and f) C M(0 + x + y) + Ua+2i3- 

Suppose that y ^ 0. If diuiH = 1, then Lemma |5.2C implies that p{h) x ||/i|n^ for h ^ H, so 
p{h) 5^ /i, which is a contradiction. Thus, we must have dim// = 2, so conclusion (3b) holds. 

We now know y = 0. If dimf) = 1 and ||x|p = —2(j)r] for some (and, hence, all) elements of f), 
then Lemma 5.14 (3b|) implies that p{h) x ||/i|p 5>i /i, a contradiction. If dim f) = 2, then Uq,+2/3 C f). 



y^ -2<A^. 



so it is easy to choose an element of f) with || 

Case 1. Assume that there is an element h of H , such that dim.{{(p,x),{0,y)) = 1, and y ^ 0. 
Because di'ni{{(p, x), (0, y)) = 1, we know that (</>, x) is a scalar multiple of (0, y), so (j) = 0. Further- 
more, we may assume, after replacing H (and, hence, h) with a conjugate under Ua, that x = 

exp{tlogh), we have 

'O(I) if i/2 andj /n + 1 
0(t) if(i,j)/(2,n + l) 



(see 3.7). Then, letting /i* 



hi, 



and ||x|p 7^ —2(1)7]. Letting 



/^^, 



and /12 „_|_i ^ t^. Therefore p{h^) x f^ x /i*, as desired. 

Case 2. Assume that there is an element h of H , such that y 
/i* = exp(ilog/i), we have 

'O(I) if i/ 1 andj /n + 2 
0{t) if(i,j)/(l,n + 2) 

and, because ||x|p 7^ —24>rj, we have /i* „+2 ^ ^^- Therefore it is not difficult to see that p(/i*) x 
t^ X /i*, as desired. 

Case 5. Assume that dvcuH = 2, Ua_|_2/3 C f), and i/iere is an element u of i) such that (/>„ 7^ and 
yu 7^ 0. Replacing i/ by a conjugate under C/g, we may assume that Xu = 0. For any large real 
number i, let h = h^ be the element of exp(tn + Ua+2(3) that satisfies r] = —(j)\\y\\'^/12. Then 



/I (/. i(/>y 



h 



1 



y 
1 



-i||y|P 
2l|y^l 

-r 
1 



\4>\\y\? 
Uy'^ 



\ 



1 



/ 



nyf- 



Because t is large, we know that 4> and ||y|| are large, so it is clear that h 

Let h' be the matrix obtained from h by deleting the first two columns. Then the first two rows 
of h' are linearly dependent (because the first row of h' is 0/2 times the remainder of the second 
row). Therefore, we have 



det 



hi^i 

h2A 



h2,j 



0, whenever i,j > 2. 



CARTAN-DECOMPOSITION SUBGROUPS OF SO{2,n) 23 

Similarly, we have 

det ( , *'" , *'" ) = 0, whenever i,j < n. 

It is easy to see that the determinant of any other 2x2 submatrix of h is O(0^||y|p). Thus, we 
conclude that p{h) = 0((^^|[y|p) = 0{h), as desired. D 

5.13. Corollary (of proof). Assume that G = S0(2, n), and let H he a closed, connected subgroup 
of N . The following are equivalent: 

1) There is no sequence {km} of elements of H with km — *■ oo and p{hm) ^^ hm- 

2) There is no curve {ht}t£M.+ in H with /ij — > oo and p{ht) ^ ht. 

3) Either 

(a) for every element of\], we have </> = and dim(x,y) / 1; or 

(b) d\m.H = 1, and every nonzero element of f) satisfies </> / and y / 0; or 

(c) dimH = 1, and every element of H satisfies y = and ||x|p = —2(j)r]. 

5B. When is the size of p{h) quadratic? 

5.14. Lemma. Assume that G = S0(2,ri), and let H be a closed, connected subgroup of N . The 
following are equivalent: 

1) There is a sequence {km} of elements of H with h^ -^ oo and p{hm) ^ ||^m|P- 

2) There is a curve {/it}tgK+ in H with /ij — > oo and p{ht) x ||/it|p. 

3) there is a nonzero element of t) such that either 

(a) (((/), x), (0, y)) is two-dimensional in W^~^ , and either dimf) /I or cp = 0; or 

(b) y = and ||x|p = —2(j)rj. 

Proof. (0 ^ §) Assume that neither (|3a|) nor (3b) holds. We have \) PI Uq,+2/3 = 0, for otherwise 
there is an element of P) for which x, y, and (j) are all zero, so (3b) holds. 



Case 1. Assume that y = for every element of f). Because P) n Uo+2/3 = 0, we know that r] is 
determined by (j) and x, so there exist p € M and b G M"~'^, such that, for every element of f), we 
have r] = p4> + b ■ X. 

Because y = and rj = 0((</>, x)), we have hij = 0((</>, x)) for all {i,j) 7^ (l,n + 2). Therefore, 
if p{h) X ||/i|P, then we must have /ii,n+2 = O((0, x)) . However, /ii,n+2 = —4'{P'P + b ■ x) — ^||x|p 
is a quadratic form on i). Therefore, because there are vectors {4>,x) with /ii,n+2 = 0[{(j),x)) <^ 
||((/), x)|p, this quadratic form must represent nontrivially, so there is an element of H with 
hi,n+2 = 0; that is, (prj + |[x|p/2 = 0, so conclusion ( ^ ) holds. 

Case 2. Assume that = for every element of f). From the negation of (^), we see that x and y 
are linearly dependent, for every element of (). Also, we may assume that there is an element of [} 
with y 7^ 0, for, otherwise. Case |l| applies. Thus, conjugating by an element of C/q,, we may assume 
that X = for every element of f) (see ^^ ). Because i) n Uq,_|_2/3, </>, and x are all zero, we know that 
r] = 0{y), so hij = 0{y) for all (i,j) / (2,n + 1). On the other hand, /i2n+i ^ l|y|P- Therefore, 
p{h) = 0{\\yf)^\\yr^\\h\\\ 

Case 3. The remaining cases. If ((/>, x) and (0, y) are linearly dependent, for every element of f), then 
either Case |l] or Case || applies. Thus, we may assume that there is some element of f), such that 
(((/), x), (0,2/)) is two-dimensional. (In particular, we must have y 7^ 0.) Then, from the negation 
of (|3a|), we see that divnH = 1, and (p ^^. Then Lemma |5.20| implies that p{h) x ||/i|p^ ^ ll^lP- 

(I =^ I) For any element h of Ua+2f3, we have hij = 0(1) for all {i,j) ^ {(l,n + 1), (2,n + 2)}, 
so it is obvious that p{h) x ||^|P- Thus, we may assume that i) n Ua+213 = 0. 
Let h £ H, and let /i* = exp(tlog h). 
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Case 1. Assume that = 0. Because i?i> = 0, it is clear that /i* • = 0{t) for all (i,i) ^ {1, 2} x {n + 
1, n + 2}. On the other hand, by hypothesis, we may assume that ((0, x), (0, y)) is two-dimensional; 

(II 'l|2 \ 
ii^p I 7^ 0, SO it is easy to see that 

\\y\\ -^'V } 



det i^,'-'^' l!i'"+'^ X t^ X \\h'f. Therefore, p(/i*) x ||/i' 

y "2,n + l ''-2,n+2 / 

Case 2. Assume that y = 0. In this case, it is clear that 



t _ J 0(1) if i / 1 and i / n + 2 



"ij 



0{t) if(i,j)/(l,n + 2) 



By hypothesis, we may assume that \\x\\^ = —2(j)r], so h\ „_(_2 = 0- Thus, because (p, x, and r] cannot 
all be zero, it is clear that p(/i*) x f^ x 11^* P- 

Case 3. Assum,e that both of (j) and y are nonzero. We see from (y) that divnH ^ 1, so there is an 
element u of H with (/>„ = 0. We know that y„ = 0, for, otherwise, [logn, [log u, log /i]] would be a 
nonzero element of f) fl Ua+2^- Then, because f) n Ua+2/3 = 0, we must have Xu / 0. 
For each large t € M, define /^ : M — > M by 

ft{s) = -t(l)h{tr]h + srju) - -\\txh + sXuW"^ + ^t^^iWyhf- 

Because t is large, we know that /t(0) > 0, so there is some s = s{t) £ M+ with ft{s) = 0. Note 
that s is approximately :^t^|0/i|||2//i||/||a;„||. 

Let /i* = exp(t log h + s log u) . Then h\ „^2 = ft{s) = 0, so we see that h^t^. Also, we have 

j„(. fK,n+l ^i,n+2\ _ r^t , t ^ f6 ^ ||;,t||2 

I A,* h^ I ~ '''l,n+l"2,n+2 ^ ^^ ^ II" II ) 

V'''2,n+1 'h,n+2j 

as desired. D 



In the statement of the following corollary, we use Proposition 5.19 to convert the condition that 
||x|p 7^ —2(1)7] to the condition that \\b\\'^ — ||c|p — 2p < 0. 

5.15. Corollary (of proof). Assume that G = S0(2, n), and let H he a closed, connected subgroup 
of N . The following are equivalent: 

1) There is no sequence {hm} of elements of H with h^ -^ cxd and p{hm) x ||/im|P- 

2) There is no curve {/it}tGM+ ^'^ H with /ij — > 00 and p{ht) >i ||^t|P- 

3) Either 

(a) for every nonzero element of f), we have (p = and dim(x,y) = 1; or 

(b) there exists a subspace Xq o/M*^"^, b G Xq, c G Xq^, antip G M i(;ii/z ||6|p — ||c|p — 2p<0, 
such that for every element of i), we have y = 0, x G </>c + Xq, and r] = pcf) + b ■ x; or 

(c) dimii^ = 1, and we have (p ^ and y 7^ for every element of H ; or 

(d) dvoiH = 1 and we have y = and ||x|p 7^ — 2(/)r/ for every element of H. 

5C. Some calculations of /i(i7). 

5.16. Lemma. Assume that G = S0(2,n). Then fi(g^^) = p{g) for all g £ G. 

Proof. This follows from the fact that there is an element of the Weyl group of SO (2, n) that sends 
each element of A to its inverse, so each element of A is conjugate to its inverse under K. D 

5.17. Lemma. Assume that G = S0(2,n), and let H be a connected, closed subgroup of N. 
Assume that for every nonzero element of (), we have = and dim(x, y) 7^ 1. Then p{h) x ||/i|P, 
for every h £ H . 
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Proof. For future reference we prove the stronger fact that for every real number t > 1 and every 
h ^ H, we have p{ah) x ||a/i|p, where 

hi, t-\t-^). 



a = diag(i, t, 1, 1, , 

Let g = ah. We have det ( ^^'^ ^^A = t^ and 

\g2,i 52,2/ 



det 



gi,n+l 9l,n+2 
92,n+l 92,71+2 



At' 



det 



hi. 



n+l 



i2,ra+l 



^l,n+2 
^2,n+2 



4t2(7y2 + 



I Il2 II ||2 

m\ \\y\\ 



(x-yf)). 
Because x and y are Unearly independent for all nontrivial h (^ H, we have 

II ii2 II ii2 / \2 II ii2 II ii2 II ii4 

||x|| \\y\\ — [X ■ y) X ||x|| ||y|| x ||x|| , 
so we conclude that p{ah) x max(t^, t^ry^, t^||j;||'^) x ||a/i|p. 



D 



5.18. Notation. We realize SO(l,n) as the stabilizer in S0(2,n) of the vector 

(0,1,0,... ,0,0,-1,0) eW^^. 

5.19. Proposition. Assume that G = S0(2,n). Suppose that H is a closed, connected subgroup 
of N and that there exist a subspace Xq ofW^^^, vectors b € Xq and a S Xq, and a real number p, 
such that 

H = {h £ N \ X e (pc + Xo, rj = p4> + b- X, y = 0}. 
If dim H > 2, then the following are equivalent: 
1) H is conjugate to a subgroup o/SO(l,n). 



2) We have \\bP 



2p<0. 



3) p{h) X h for every h £ H . 

4) We have ||x|p ^ —2(f)ri, for every nonzero element o/f). 



Proof (I ^ I) Suppose that ||6f - | 
real number. Set 

/t 



2p < 0. Then t := \/— (||&|P — ||c|p — 2p) is a positive 



9 







^-l +-1 



t-^{c-b) 




v 










0\ 


}t- 


i||c- 


bf 





{c 


-by 









t-' 








Then a matrix calculation, using the facts that c ■ b = and c • Xq = 0, verifies that y = and 
T] = (j) for every element of gHg~^, so gHg^^ C SO(l,n). 

(HI ^ P) Because SO(l,n) is reductive and A^ n SO(l,n) is the positive Weyl chamber of a 
maximal split torus of SO(l,n), we have /u(SO(l,n)) C A'^ n SO(l,n). Then, because 

AnSO(l,n) = {diag(a, 1,1,... ,l,l,a"^) | a > 0}, 

it is clear that p{g) x g for every g G S0(1, n). 
(3 =^ 4) This is immediate from Lemma ^.14|(|3b ). 
(4 =^ 2) We prove the contrapositive. 
Assume for the moment that 6^0. The equation \\sc + b\\^ ■ 



-2s {ps + 



is quadratic 



(or linear) in s. Because the discriminant 4|[6|p(||6|p — ||c|p — 2p) of this quadratic equation is 
nonnegative, the equation has a real solution s = (J)q. Then, for the element of i) with (p = (J)q, 
X = </>oc + b, y = 0, and r/ = p(j)Q + ||6|p, we have ||x||^ = —2(j)r]. Because b ^ 0, not both of (j) and x 
can be 0, so we have the desired conclusion. 
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We may now assume that 6 = 0. Because dimH > 2, there is a nonzero element xq of Xq. 
Because ||c|p + 2p = — (||6|p — ||c|p — 2p) > 0, the equation \\sc + xqP = —2s{ps) has a solution 
s = (j)Q. Because xq ^ 0, we know that (pQ ^ 0. For the element of \) with (p = (pQ, x = (pQC + xq, 
and r] = pcpo, we have ||x|p = — 2(/)r/. D 

5.20. Lemma. Assume that G = S0(2,n). Let H be a closed, connected subgroup of N, such that 
dim// = 1, and we have (/> 7^ and y 7^ for every element of H. Then p{h) x ||/i|p^, for every 
heH. 

Proof. Let /i* = exp(tn), where u is some nonzero element of i). For any large real number t, we 
see that /^i „+2 ^ ^^> but /i* • = O(t^) for (i,j) 7^ (l,n + 2). Thus, /i* x f^. Furthermore, we have 

i,t'"^^ !,t'"^^ ) ^ *^) ^-iid we have /i* • = 0{t'^) whenever i 7^ 1 and j 7^ n + 2, so we conclude 

"-2,71+1 "2,n+2 J '-' 

that p(/i*) X t^ - ||/i*||3/2^ as desired. 

For future reference, we note the stronger fact that for every real number s > 1 and every h £ H, 
we have p{ah) x ||a/i|p^, where 

a = diag(s^,s,l,l,... , 1, 1, s"\ s"^). 

D 

6. Subgroups of S0(2,n) that are not contained in A^ 

6.1. Theorem. Assume that G = S0(2,n). Let H be a closed, connected subgroup of AN , such 
that H = {H n ^4) K [H n N). The subgroup H is a Cartan- decomposition subgroup of G if and 
only if either 

1) Ad H; or 

2) H n N is a Cartan- decomposition subgroup of G {see |5.j| ); or 

3) H n A = ker a, and we have (p = rj = and dim(x, y) = 1 for every h € H f] N; or 

4) H n A = ker P and we have y = and ||x|p = —2(pr] for every element of H n N; or 

5) there is a positive root u such that f) n n C u^^j, and H satisfies the conditions of Proposi- 



tion 3.11. 



6.2. Corollary (of proof). Assume that G = S0(2,n). Let H be a closed, connected subgroup 
of AN, such that H = {H D A) K {H f] N), and H c/L N . The subgroup H is not a Cartan- 
decomposition subgroup of G if and only if either 

1) H = H n A is a one- dimensional subgroup of A; or 

2) H n A = ker a, and we have = and dim(x, y) 7^ 1, for every nonzero element of i) rin, in 
which case p{h) x ||/ip for all h G H; or 

3) H Pi A = ker (3, and we have (p = 0, y = 0, and x ^ 0, for every nonzero element of i) Dn, in 
which case p{h) x h for all h £ H ; or 

4) H r\ A = ker(a + (5), and we have (/> = 0, x = 0, and y ^ 0, for every nonzero element of 
f) n n, in which case p{h) x h for all h G H; or 

5) HnA = ker/5, and there exist a subspace Xq ofW^^"^, b G Xq, c G Xq, and p G M, such that 
II^IP ~ l|c|P — 2p < 0, and we have y = 0, x G (/>c + Xq, and r] = pep + b ■ x for every h £ H, 
in which case p{h) x h for every h G H; or 

6) H n A = ker(a — f3), diuiH = 2, and there exist (/> G Uq and y G Up, such that p ^ 0, y ^ 0, 
and f) n n = M(i?!) -\- y), in which case p{h) x ||/i|P'^ for every h G H; or 

7) H r\ A = ker/3, divnH = 2, and we have y = and ||x|p 7^ —2<pr] for every h £ H, in which 
case p{h) x h for every h G H; or 

8) there is a positive root uo and a one- dimensional subspace i of a, such that f) n n C u^, 
f) = t+ (f) nn), and Proposition 3.1% implies that H is not a Cartan- decomposition subgroup. 
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Proof of Theorem |6.1| . We may assume that fiPin is not contained in any root space, for otherwise 



Proposition 3.17 apphes. We may also assume that H (^ A, and that H r\ N \s not a Cartan- 



decomposition subgroup. The proof proceeds in cases, suggested by Theorem |5 

Case 1. Assume that (p = {) and that dim(x,y) 7^ 1, for every nonzero element 0/ f) fl n. We show 
that p{h) X ||/i|p, for every h G H, so H is not a Cartan-decomposition subgroup. Let h G H, 
and write h = au with a & H f) A and u € H n N. Because f) n n ^ Ua+2i3 (recall that f) n n is 
not contained in any root space), we know that t) n n projects nontrivially into U/3 + Ua+p- This 
projection is normalized by H A, but, from the assumption on dim(x,y), we know that this 
projection intersects neither U/3 nor Ua+i3, so we must have a{H DA) = 1. Thus, a is of the form 
a = diag(t,t, 1, . . . , l,t^^,t^^). We may assume that i > 1, by replacing h with h^^ if necessary 
(see 5.161 ). Then the proof of Proposition [5.17 implies that p(h) > 



Case 2. Assume that = and that dim(x, y) = 1, for every element 0/ f) Pin. We show that i7 is a 
Cartan-decomposition subgroup if and only [ia{HnA) = 1 (and f)nn C Uf^ + Ua+p)- Furthermore, 
when H is not a Cartan-decomposition subgroup, we show that p{h) x h for every h & H. 
(<^) Because a{H f] A) = 1, we know that every element of i? n A is of the form 



diag(t,t, 1,... ,l,t"\t"^), 



so p(a) X t X ||fl|| • From Lemma 5.12| we also know that there is a path h ^^ 00 in H PiN with 



p(/i*) X /i*. Thus, H is a Cartan-decomposition subgroup. 

(^) Suppose that a{H nA)^l. Then, because dim(x, y) = 1 for every h G H n N, we see that 
\} n Ua+2f3 = and either 

i) 1} n n C Ufs + Ua+2f3; or 

ii) i}nnCUa+f3+Ua+2f3- 

Because the Weyl reflection corresponding to the root a fixes Ua+213 and interchanges Up with Ua+p, 
we may assume that (||) holds. Then, by choosing a large negative value for p, we see that H is a 
subgroup of a group of the type considered in Case ^ below, so p{h) x h for every h £ H. 

Case 3. Assume that there exist a subspace Xq of R"~^, b G Xq, c G ^0 ; flf^c^ p E M, with 
II^IP ~ l|c|P — 2p < 0, such that we have y = 0, x £ (j)c + Xq, and rj = pcj) + b ■ x for every h £ H. 
We show that H is conjugate to a subgroup of SO(l,n), so p{h) >; h for every h £ H. Therefore, 
H is not a Cartan-decomposition subgroup. 

Let h £ H, and write h = au with a G /7 n j4 and u £ H f] N. Because y = for every element 
of f) n n, we know that i) Hn C Ua + Ua+f3 + Uo+2/3- Also, because ||6|p — ||c|p — 2p < 0, we know 
that p and c cannot both be 0, so f) n n does not intersect Uq, and, because rj = p4> + b- x, we know 
that [) n n does not intersect Uq,+2/3- Thus, because f) n n ^ Ua+/3, and i7 n A normalizes f) fl n, 
we see that P{H n ^4) = 1, so iiZ" n ^ C S0(1, n). Furthermore, H (1 A centralizes U/3, so H (1 A is 



normalized by the element g in the proof of Lemma 5.19 that conjugates ^ n A^ to a subgroup of 



SO(l,n). Thus, we conclude that g conjugates all of // to a subgroup of SO(l,n). 

Case 4- Assume that dim{H D N) = 1, and that (p ^ and y 7^ for every nontrivial element 
of H n N. We show that p{h) x ||/i|p^ for every h £ H, so H is not a Cartan-decomposition 
subgroup. 

Because H f] N projects nontrivially into both u^ and u^, but does not intersect u^, we must 
have a{a) = (5{a) for every a £ H r\ A; that is, a is of the form a = diag(t^,t, 1, . . . , l,t~-^,t~^). 
Because dim(i/ n A^) = 1, there exist (j) £ Ua, x £ Ua+p, y £ Up, and fj £ Ua+2/3, such that 
f) n n = W{(j) + X + y + fj). However, because (/>-|-x-|-y-|-r/isan eigenvector for every element of 
ker(a — /3), but the restriction of ker(a — /?) to Ua+p or Uq,+2/3 is different from the restriction to each 
of the other root spaces u^ and U/3, we see that x and fj must both be zero. Thus, f) n n = W{(j) + y) . 
From the proof of Lemma 5.20| , we conclude that p{h) x ||/i|p^, as desired. 
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Case 5. Assume that dim.{H N) = 1 and that i) On is not contained in any root space. We show 
that H is a Cartan-decomposition subgroup if and only if either 

1) we have cp = 0, -q = 0, and dim(x, y) = 1 for every element of f) n n; or 

2) we have y = and ||a;|p = — 20r/ for every element of H n N. 

We also show that if H is not a Cartan-decomposition subgroup, and neither Case |l|, Case P, nor 
Case Q applies to H, then we have p{h) x h for every h ^ H. 

We may assume that (/> 7^ 0, for some (and hence every) nonzero element of f) n n, for otherwise 
Case H or |2| applies. Then we may assume that y = 0, for otherwise Case Q applies. 



If ||x|| = — 2(/>r7, then Lemma 5.14 implies that p{u) x \\u\\ for every u ^ H Ci N. Furthermore, 



because P{H n ^4) = 1, we know that p{a) x a for every a £ H D A. We conclude that H is a 
Cartan-decomposition subgroup. 

On the other hand, if ||x|p 7^ — 2(/>r/, it is not difficult to see that p{h) x h for every h a H 
(cf. Case I of (| ^ |2|) in the proof of Lemma |5.12| ). D 



We now consider the case where H is not a semidirect product of the form T \kU , with T <Z A 

and U C N. 

6.3. Theorem. Asswne that G = S0(2,n). Let H he a closed, connected subgroup of AN that 
is compatible with A {see [g!^ ). Assume that H ^ {H D A) t< {H tl N) and that H (1 N is not a 
Cartan-decomposition subgroup. The subgroup H is a Cartan-decomposition subgroup if and only 
if H is abelian and there exists to £ {/?, a + /?} such that A n (HN) = kerw, f) n u^^ 7^ 0, and 
i) Ci + Uuj + Uq+2/3) where i is the Lie algebra of Ar\ {HN). 

We now describe the Cartan projections of those subgroups that are not Cartan-decomposition 
subgroups. 

6.4. Corollary (of proof). Assume that G = S0(2,n). Let H be a closed, connected subgroup 
of G that is compatible with A {see \2.!^ , and assume that H ^ {H n j4) x {H n A^). Then there is 
a positive root uj, and a one- dimensional subspace y of (kercj) -|- u^^, such that f) = p -|- (f) n n). 

If H is not a Cartan-decomposition subgroup ofG, then either: 

1) u = a and f) n n C U0+/3, in which case p.{H) « [||/i||, ||/i|p/(log \\h\\)\ ; or 

2) u) = a and f) n n C Uo+2/3, in which case p{H) « [||/i|p/(log |[/i||)^, |I/i|P] ; or 
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or 



3) u) = a + 2j3 and () n n C Uq, in which case p{H) ^ [||/i|p/(log ||/i|l)^, \\h\ 

4) ll> = a-\-2l3 and either f)nn C u^ or ijHn C Ua+i3, in which case p{H) « [||/i||, ||/i|p/(log \\h\\)\ ; 
or 

5) cj S {l3,a + 13}, f) n n C U(j + Ua+2(3, o,nd f) n u^^ = f) n Uq+2/3 = 0, in which case p{H) '^ 
[WhUhfl^]; or 

6) there is a root 7 with {uj, 7} = {/?, a + 13} , f) n n C u-^ -|- Ua+2/3, and f) fl Ua+2/3 = 0, in which 
case^,{H)^[\\hl\\h\\{\og\\h\\f]■,or 

7) UJ £ {P,a -\- /?} and f) n n = Uq,+2/3; in which case p{H) ?a [||/i||(log \\h\\), ||/i|P]; or 

8) UJ = a + (3 and f) n n = Uq,, in which case p{H) « [|[/i||(log ||/i||), ||/i|P] • 



Proof of Theorem 6^. Let T = Ar\ {HN). There is a positive root uj with uj{T) = 1, and a 
one-parameter subgroup W of [/^, such that P)Ct-l-tt)-l-(f)nn), where It) is the Lie algebra of W 
(see P^ ). Note that T and W are contained in the Zariski closure of H. 

It is not difficult to see that every nonabelian subalgebra of n contains Ua-|_2/3. Therefore, if \) 
does not contain Ua+2/3, then P) n n is abelian, and is centralized by W . 

The proof proceeds in cases. If diui{H n A^) > 2, then Theorem |5.5| implies that one of Cases |l], 
Q, or |5| applies. If dim(if n A^) = 1, then there are many more possibilities to consider. Let us see 
that none of them have been overlooked. Because of Case ^, we may assume that f) n n ^ v^a+2i3, so 
if = in f) n n, then either Case || or Case ^ applies. Thus, we may assume that (/> 7^ in f) n n. 
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Because of Case ^ we may assume that f) n n 7^ u^, so t) n n is not normalized by A. This imphes 
that u) = (3, and f} n n C Uq, + Ua+fs + Uq,+2/3, so Case || apphes. 

Case 1. Assume that f) n n ^ ^0+2/3, CLf^d that we have = and dim(2;,y) 7^ 1, for every nonzero 
element 0/ f) n n. We show that this is impossible. Because f) n n ^ Uq,+2/3 and dim(a;, y) 7^ 1, there 
is some h a H with both x and y nonzero. Since T normalizes HCiN, we must have a{T) = 1 (that 
is, uj = a), so the Zariski closure of H contains Ua- We have [h,Ua] C Ua+i3Ua+2f3, but, because 
y 7^ 0, we know that [h, Ua] <^ Ua+2p- Thus, f) n n contains an element of the form x + 7), with 
X € Ua+f3 and 17 G Uq,+2/3, such that x 7^ 0. This contradicts the assumption of this case. 

Case 2. Assume that uj = a. We show that K is not a Cartan-decomposition subgroup. Fur- 
thermore, f) n n is contained in either Ua+/3 or Ua+2/3! and \i{ii) k. [||/i||, ||/i|p/(log ||/i||)^] or 
iiiU) « [||/i||V(log||/i||)M|/if], respectively. 

Suppose that (lt),f) fin) is nonabelian. We must have Uq,+2/3 C f). Then, since (rD,UQ,+2/3) is 
abelian, we must have dim(f) n n) > 2. 

• If Conditio n |5.5|(|2| ) holds, then Case |l] applies. 

• Conditions ^.5|(^and 5^(^ cannot hold, because Uq,+2/3 C f) n n. 

We conclude that ff n A^ is a Cartan-decomposition subgroup, which contradicts a hypothesis of 
the theorem. 

We now know that (ro, i] Pin) is abelian. We must have f) Hn C Uq, + Uq,_|_/3 +Uq+2/3) the centralizer 
of Uq, in n. We know that ker a normalizes f) n n, that the restrictions of a, a + /?, and a + 2/3 to 
ker a are all distinct, that Uq, ^ f) n n, and that H Ci N is not a Cartan-decomposition subgroup, so 
we see from Proposition 5^ that f) n n must be contained in either u^+zj or Ua+2p- 

Subcase 2.1. Assume that t) n n C Ua+p. We have 



/e* 



iJ 



where Xq is a subspace of I 

We may assume that t, 
2/11 I 



te' 



e*x 

Id 









te 



\x\ 




/2\ 


' 




t GR, 




XGXo 


/ 





x 



' and 00 is a nonzero real number. 
> 1 (see 5.16 and 3^). Then h 



max{te*, e*||x|p} and p{h) 



letting t 



Letting t = 1 yields p{h) 



h. The largest relative value of p{h) is obtained by 



which results in p{h) x te x j|/i|| /(log 



Subcase 2.2. Assume that f) n n C Uq,_|_2/3. We have 



H 



/e* 



v 



te*0o 





Id 








-te^(j)orj\ 



-e^rj 





e" 



/ 



t,?7G 



e^*?7^. 



that det 



0.) Letting t = 1 yields p(/i) x r/ 



(In calculating ||/9(/i)||, one must note 
^ X II ^IP- The smallest relative value 

o2t ^ 



where 0o is a nonzero real number. 

Assuming t, |r/| > 1, we have h x te*|ry| and p{h) 
hi,2 hl,n+2 
Jl2,2 h2,n+2 

of p{h) is obtained by letting ?y = 1, which results in p{h) x e^* x ||/i|p/(lo, 

Case 3. Assume that u; = a + 2(3. Since the restrictions of a, (3, a + (3, and a + 2/3 to ker(a -|- 2/3) 
are all distinct, and H n N is not a Cartan-decomposition subgroup, we see from Proposition ^ 
that f) nn must be contained in either Uq,, u^g, or Ua+p- Because the Weyl reflection Wa interchanges 
Up and Uq,+/3, we may assume that f) n n is contained in either Uq, or up. 
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Subcase 3.1. Assume that f) n n C Uq,. We have 



H 



r /e* 



I V 





Id 








te*0%\ 


te-^rjo 





-eV 


e- y 



*,(/)£ 



where r/o is a nonzero real number. 

Assuming that t, \(p\ > 1, we have h x ie*(/) and /9(/i) x e^'^cjP'. (Note that det 



Letting t = 1 yields p{h) x 
(^ X 1, which results in pQi) x e 



/i2,2 



/il,n+2 
/i2,n+2 



0. 



^. The smallest relative value of p{h) is obtained by letting 

2*^ll/l||V(log||/l||)2. 



Subcase 3.2. Assume that f) n n C u^j. We have 





r /e* 



H 



where Xq is a subspace of 




Id 



te*r/o 







eix||V2\ 
-te-^Vo 

— X 




e- y 



pn— 2 



and r]Q is a nonzero real number. 
Assuming t, \\x\\ > 1, we have h x max{e*||x|p,ie*} and p{h) 



t gM, 
X e Xq 



2t It I 

ex 



p{h) X I 
in p(/i) X te 



2t 



/i. The largest relative value of p{h) is obtained by letting t 
V(log 



Letting t = 1 yields 
< llxlP, which results 



Case 4- Assum,e that we have = and dim(x,y) = 1, /or every nonzero element of [} On. Because 
dim(x, y) = 1, we know that f) does not contain Ua+2p- Therefore, f)nn is abelian, and is centralized 
by the one-parameter subgroup W ol Uuj. 

We may assume that a; G {/?, a + /3}, for otherwise Case ^ or ^ applies. 

Note that either x = for every h(^Hr)N,oTy = for every h (z H N. (If there is some 
h G H n N with both x and y nonzero, then, because uj ^ a, both f) n u^ and f) n Ua+f3 must be 
nonzero. Then, since dim(x,y) = 1, we see that f) n n is not abelian, which is a contradiction.) 
Because the Weyl reflection corresponding to the root a fixes Ua+2/3 and interchanges U/3 with Ua+/3, 
we may assume that y = for every h G H f] N. Thus, f) n n C Ua+i3 + Ua+2/3- 



Subcase 4-1- Assume that uj - 

/I 



a + p. We have 



H 







txo + X 

Id 



bo -x 





V 



-||txo + x|lV2\ 

— e*5o • X 

-(txo + x)^ 



1 / 



teR, 

X G Xq 



where Xq is a subspace of M""^, and 69 and xq are fixed vectors in M"^^ with xq ^ Xq. 

Because xq ^ Xq, we have ||txo + x|| x max{|t|, ||x|j}. Assuming that t, ||x|[ > 1, we have 
h X max{e*, ||x|p, e*|feo • x\} and p{h) x maxjt^e*, e*||x|p}. Letting t = 1 yields p(/i) x ||x|p x /i. 

If there is some nonzero xi G Xq with 60 ' ^^i = 0, then letting x = xi and e* = |lx|p yields 
p{h) X ||x|j^ X ll^lP; so if is a Cartan-decomposition subgroup. 

On the other hand, if we have 60 • x 7^ for every nonzero x G Xq, then 60 ' 2; >i x for all 
X G Xq. The largest relative value of p{h) is obtained by letting e* x ||x||, which results in 



p{h) 



,3t 



\h0\ 
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Subcase J^.2. Assume that uj = 



H 



(3. We have 




e^x e*6o ■ x 

1 tyo -t'\\yofl2 
Id -ty^ 
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^11 Il2 

— e x / 
-bo -x 





V 



/2\ 



/ 



t eM, 
X eXo 



on— 2 



where Xq is a subspace of M" , and 6o and yo are fixed vectors in 
Assuming i, |[2;|| are sufficiently large, we have h x e*||x|p and p{h) 



with 2/0 7^ and yo ^ -'^o- 
. Letting t = 1 



t^e*! 



yields />(/i) x 
results in /9(/i) x t^e 



2 A 



h. The largest relative value of p{h) is obtained by letting 



1, which 



og 



\h\ 



Case 5. Assume that fiPin C u^ + Uq+^ + Uq+2/3 > that fjnn ^ ^a+ii + '^a+2p, that Uq,+2/3 ^ ^Hn, and 
that f) nn is noi normalized by A. Because f) Pin is not normalized by A, but is normalized by T, we 
must have P{T) = 1, so a; = /?. Also, because Uci+2/3 ^z! f) Pin, we know that f) nn is centralized by to. 
Because the centralizer of tr (or any other nontrivial subspace of u^) projects trivially into u^, this 
is a contradiction. 

Case 6. Assum,e that f) n n = Uq+2/3- We may assume that to G {/?, a + /?}, for otherwise a; = a, so 
Case ^ applies. Then, because the Weyl reflection Wa interchanges /3 and q + /3, but fixes a + 2/3, 
we may assume that cv = a + p. We have 



H 



/I 



V 



teo 


T] 








Id 






-i'lkolPM 



-e*?7 

-tXr, 



e"* 

1 

Assuming t, |7?1 

2 ^ 111,112 



t,r/G 



t^e* 



> 1, we have h x e^\rj\ and /9(/i) x 
The smallest relative value of p{h) is 

og\\h\\). 



where xq is a nonzero vector in M""" 
max{t^e*, e*7/'^}. Letting t = 1 yields p(/i) x r^ 
obtained by letting t] '^ t, which results in p{h) 

Case 7. Assume that i) (In = Ua- We may assume that iv £ {/3, a + /?}, for otherwise co = a + 2/3, 
so Case ^ applies. Since Uo+2/3 ^z! f) n n, we know that tr centralizes f) fl n. Then, because Uq, does 
not centralize any nontrivial subspace of up, we conclude that uj = a + p. We have 



H 



/I 



v 



txo 





-t^Xo 











Id 





— tx 



V2\ 



t,^e 



where xq is a nonzero vector in M"~^. Assuming t, 
max{(/!)^/e*,i^e*}. Letting t = 1 yields p{h) >: (j? -^ 
obtained by letting cp >: te^, which results in p(h) x 



> 1 

2 



t^e* 



we have h x max{e*, \(j)\} and /9(/i) x 

The smallest relative value of p{h) is 

|/i||log||/i||. D 
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